How a vendor's security negligence gave me root access to ticket vending machines across the US

This post details my experience with companyX, a popular vendor that sells ticket vending machines (referred to as TVMs) for bus agencies. This story is meant to educate others about the danger and risk associated with letting vendors run amuck without checking for security flaws. In my personal experience, most vendors I've worked with do not take security seriously.

As the title suggests, the research done here is by no means sophisticated and was super easy to conduct.

These bastards did not like the idea of public disclosure, so all information regarding the specifics have been removed or redacted.

The backstory

I've worked with this particular vendor in the past and have been frustrated with their deployments, maintenance, and general support. I've been skeptical about the vendor as a whole's competency regarding their systems, and especially their product's security.

This particular vendor has security monitoring tools. I'd avoid that software with a 10 foot pole after learning about this. Yikes!

These TVMs are often deployed at remote sites, with an LTE modem and router. In order for these TVMs to get polled for fare statistics and have remote access, certain ports are port forwarded to the internet.

Some time ago, I learned that their TVMs were configured with default credentials that consisted of a three letter username, with the same three letter password. I warned the vendor that this was extremely bad practice and the support tech laughed. I later instructed my coworker to request this formally, and they were told that their maintenance scripts rely on these credentials to work. According to the bash history, the password was set back to the default credentials.

As if that's not disturbing enough, these credentials have sudo access as well. In a nutshell, you can get root access by knowing their default three-letter credentials!

Some time passed by and I had forgotten about this (oops) until one day, I was reading my recently set up cyber hygiene scan results. (Thank you CISA!) It wasn't until their report showed me the operating systems these devices were running and I was completely shocked. It was rated a 10/10 severity to no surprise.

Without divulging too much information, here is the banner that shows the operating system and version of OpenSSH running on the TVM:

SSH-2.0-OpenSSH_4.7p1 Debian-8ubuntu1.2

This indicates the OS is specifically running Ubuntu 8.04.1 LTS. That OS was released in July '08 and was last supported until May 2013. That means that these TVMs have not received security updates for nine years. When I had asked about the historical knowledge about the deployment of these devices, it seems like they have been out in operation with no security updates for this entire time. When I first looked into this years ago, I naively assumed this would be something that would be addressed at some point. Considering that was not the case, I knew it was only up to me to demonstrate how bad this situation was.

TVM hunting

Considering the TVMs deployed at my organization had those awful default credentials, my curiosity piqued and I wondered if other customers had the same configuration. To start with, I searched my TVM's IP address on Shodan to get a baseline of what to expect when looking for others. I next searched the OpenSSH banner from above and was presented with 413 results in the US. I narrowed the scope to the US because I don't believe this vendor has customers outside the country. Roughly half of those results belonged to various cloud hosting companies, so they can be deductively ruled out because ticket vending machines are hardware and not hosted in the cloud.

The remaining results were potential TVMs and the hunting began!

From here, I started looking at the specific Shodan results and found something huge - some of these TVMs are configured with mDNS on port 5353. On the mDNS banner, it clearly spelled out:

mDNS:
services:
9/tcp workstation:
Name=tvm3 [00:50:00:50:00:50]
Address=1.2.3.4 fe80::250:ffff:ffff:ffff
answers:
PTR:
_workstation._tcp.local

Notice the Name=tvm3? Dead freakin' giveaway it's a TVM.

From here, I knew with confidence this had the same default credentials. After going through the results manually, I was able to compile a list of TVMs across the US.

I noticed that a majority amount of these TVMs were located in one city in particular. That city's bus agency happens to be this vendor's biggest customer with at least a dozen of these TVMs deployed. More about that later.

I took the "Name=tvm3" string from the banner and ran another Shodan search using it. From there, I was able to find a few more TVMs that belong to other agencies.  I knew there were more to find...

Curiously, I wondered if these TVMs are provisioned using a golden image. I copied the public SSH key shown on the banner and used it to run another search query on Shodan. To my surprise, I was right:

Using this search, I was able to find every single TVM belonging to this vendor on the internet. The final number ended up at 49. That means 49 ticket vending machines are deployed in the wild, sitting there with nine years worth of vulnerabilities, processing debit/credit cards, with customers none-the-wise.

White hat? But the opportunities to be naughty!

Having root access to 49 ticket vending machine isn't too much to brag about, especially if I didn't even have to launch metasploit. Sure, there were many malicious things I could do with them, including the classic ransomware attack, but these devices belong to transit agencies across the US whom probably already struggle with network security. I fault the vendor, as it's 100% their fault for deploying systems with completely insecure configurations and lack of system hardening.

Going back to the struggle with small agencies with one overworked admin, it should not be completely up to a sole person to have the sophisticated understanding this deep. As the sole systems administrator at my organization, I definitely feel for those in similar situations. We are expected to be the knower of all things, which can be overwhelming. How can a single employee be responsible for countless servers, workstations, network equipment, and auxiliary sites with hardware?

With this being said, the ethical choice was obvious. The large city's bus agency from above was first on my list as they had the most deployed. Getting a hold of someone who could understand me was the challenging part.

I'm not hacking you!

The first time I attempted a cold call, I was quickly dismissed when trying to explain the nature of my call to the dumb naive helpdesk employee. "I'm hanging up now...[click]"

I don't blame the guy for being skeptical. But I do blame him for being a moron the second time I called. This time around, I was able to get a direct number to reach their helpdesk line. I happened to reach the same person from the previous call, which was funny. This time, I started off the call by slowing introducing myself, provided my name, title, place of work, phone number, etc.. I was put on hold for a while, then finally, "Can you please repeat what you were saying?" I was obviously on speakerphone with others listening, so I repeated my spiel and alerted them with my findings.

"Okay, we'll put you in touch with the right contact for this. We'll call you later."

"Later... later as in today, tomorrow, next week?"

".... today"

The call never came that day. However, the next day, I arrived at work with four missed calls from what I assumed was this contact. Turns out, it was actually the regional field office for the FBI. I patiently waited for another call until the agent called again (this time through our helpdesk's general number) and let me advise him on my findings. The agent was cool with me after understanding I was attempting to alert this organization. He had mentioned a misunderstanding a few times but never elaborated on what they were suspecting me of. After attempting to verify myself by providing my name and employement, he still did not believe me 100%. He did admit though, that I picked up the phone on the queue and that did show I was somewhat legit. I also provided him with my driver license and it still wasn't good enough! What I find comical is that he asked to speak to my boss and have him vouche for me. It could have been joe-blow off the street, but somehow this was enough to convince the nice agent that I am indeed legit and practice network security as a grey white hat. The agent ended up thanking me for helping and said he would have the organization reach out to me to get the information to the appropriate contact.

Shortly after, I get a call from a nice network engineer belonging to the agency. We spoke at length about our frustrations with this vendor and how his mitigation attempts weren't working. He mentioned that he had also been aware of these security flaws based on Nessus scans performed for his agency. He was very appreciative about my offer to help and thanked me countless times. I told him that I would follow up with him, after my meeting with the vendor the next day. Toward the end of my conversation with him, I told him I was curious about what their helpdesk thought I was trying to say and that I had been contacted by the FBI. He said that he blamed his helpdesk for flipping out and assuming I was trying to "hack their TVMs" ?. He assured me he wasn't trying to get me in trouble, which I appreciated.

But it begs the question: Why on earth would I provide all of my contact details including place of employment if I was trying to "hack them????" It takes a person with no common sense skills to arrive to that conclusion, but hell, whatever right? No harm no foul!!

Meeting with the vendor

Before jumping into all the details with the vendor, I wanted to hear them admit they had no current OS lifecycle for their devices. When I had explicitly asked them about it, they tiptoed around answering the question, stating that they were adding lifecycles to their new products going forward. When I asked again about the lifecycle of their specific tvm3 model, they once again wouldn't go into detail. I knew instantly that my suspicions had been correct. Regarding updates, they played stupid and said they "have software updates often." Sure, they are updating little things here and there, but by no means are they actually updating the OS or packages running on the system.

After detailing my findings, I was surprised that no one from the vendor team asked to see proof of any sort. They later asked for me to provide the specifics regarding vulnerabilities, but no PoC, live demo, or anything.

I then went into my mitigation advice, and they were all silent. I definitely gave them great advice, but they asked for me to compile this information for them and provide it after.

As if it wasn't embarrassing enough, they expected free security advice from the guy who just pointed out their security flaws! Part of me would have loved to tell them to fuck off and figure it out themselves, considering they were boasting their parent company was "worth 3 billion dollars."

Can I publicly disclose this and get credit?

"We'll need to run it by legal."

Well duh of course. But where I was pissed off, is when this person told me they have a community forum for customers and this would be welcome there if approved. As if I give a fuck about their stupid forum.

I understand that it's difficult to take responsibility after just getting your security pants pulled down like this, but of all the people in that meeting, not one person expressed their gratitude or appreciation for the work I had done. I wasn't asking for a monetary bounty or even swag - just the credit and ability to disclose this information outside of their cute little forum.

So it was fixed, right?

That meeting was on 7/23/21. Since the meeting, the project manager asked me to go into further detail on my advice twice now. On the second time, I was once again surprised how incompetent their group was, how no one in their staff had a clue about OS hardening, or any network security practices in general. Here's a snippet of one of my responses to them, after providing free advice like a goddamn chump for the last time:

"I apologize for being blunt, but I would highly recommend you look into contracting a security professional to make a final determination on your mitigation and possibly have security audits every few years."

For the record, I am not this nice to negligent morons like this. The only reason why they got the nice version of me is because I was stupid enough to present my findings on the behalf of my organization. I already had one exec complain about my "unprofessional-ism" when I had asked for credit regarding my findings. I won't bother going into that.

Anyway, if you'd probably expect by reading this far, they haven't mentioned a single thing regarding their mitigation strategy, lifecycle of this old fleet of TVMs, or anything for that matter. I asked my contact over at our sister agency and they also reported radio silence from the vendor. At the time of writing this, it's almost been a month since first reported and I frankly don't expect an update for another month, if that.

So it was fixed, right?????

No.

WTF!?

I finally got a response back on 9/7/21. They didn't even bother CC'ing me, I had a coworker to forward it. The vendor used this as on opportunity to up-sell us to more of their crappy systems. The same systems that will be end-of-life by 2024. Considering these TVMs have been deployed for more or less than ten years, it would be extremely foolish to drop serious cash on something that will be not supported in a few years, continuing the cycle of legacy garbage in production.

To paraphrase them, it was along the lines of "We've listened to your concerns and established lifecycles for newer models going forward. Here's the pricing to retro-fit your model. We advise you upgrade to remain safe."

On top of that, their advice to migitate my findings was my own advice I had provided them, just watered down in their words. I'm absolutely shocked they would have the audacity to do this and not even provide me with the credit.

At the end of this, they walked away with a free vulnerability assessment, mitigation advice, and even engineering procedure advice. Doing the right thing clearly was not the correct approach, as the vendor made it explicitely clear they are not interested in the security of their products, their customers, and only care about using this as an opportunity to sell us more garbage systems.

End of the story and lesson

This story ends on a disappointing note, which is that absolutely nothing changed or will be changed due to this ridiculous security hole. The vendor doesn't care, my agency doesn't care, and basically no one gives a shit that we have systems in place that are way past legacy. Considering I've done my due diligence on ethically reporting, it's now up to management to decide on how to proceed. If I were to make an educated prediction, absolutely nothing will result from this.

The lesson of this story is that the white-hat approach is flawed for the reason of capitalism, pure and simple. The vendor could make an effort to help get their customers on newer versions, but instead they are upselling in pursuit of the almighty dollar bill. My agency, along with the numerous others, will continue to keep these systems deployed and force sysadmins like me to support them because it simply is too costly to upgrade them.

In hindsight, the better approach would have been to cryptoware them and force the vendor to fess up and scramble to figure out things on their own. It would have been a good lesson for them to learn why it's not a good idea to deploy systems for decades on end without any sort of security baked in. These types of companies need the "nuke and rebuild" approach because their engineering decisions were too far fucked to fix. I would feel bad for customers who couldn't purchase fare media, but the applicable saying for this goes, "You have to break some eggs to make an omelette."

EOM