I’m working on a some materials for a class wherein I’ll be teaching some young, wide-eyed Windows nerds about Linux and we’re including a section we’re calling “foot guns”. Basically it’s ways you might shoot yourself in the foot while meddling with your newfound Linux powers.
I’ve got the usual forgetting the . in lines like this:
$ rm -rf ./bin
As well as a bunch of other fun stories like that one time I mounted my Linux home folder into my Windows machine, forgot I did that, then deleted a parent folder.
You know, the war stories.
Tell me yours. I wanna share your mistakes so that they can learn from them.
Fun (?) side note: somehow, my entire ${HOME}/projects folder has been deleted like… just now, and I have no idea how it happened. I may have a terrible new story to add if I figure it out.
I wanted to try inserting and removing kernel modules, so I looked around and thought “well, I don’t have a USB stick in right now so I can safely try removing the usb kernel module.” So I did that, and after pressing enter I realized my keyboard is connected with USB.
I was smug thinking “I haven’t done anything so silly as the people commenting in this thread”, then I came across this one. I’ve actually done this one, and it was earlier this year, and I’ve been using Linux since 2004, 20 years.
Running the right command on the wrong SSH session/machine.
This is the scariest comment I’ve read in this thread.
Imagine that you’re using fqdns instead of ips…
I refuse to elaborate.
I set a different background color on all my machines because of exactly this while using VNC/RDP
Neat idea, I hope I’ll remember this when I’m setting up my next server.
this. after i set different zsh themes on my servers + my main machine i now know exactly what machine i’m running commands to
Me every single day
An older friend of mine told me years back about an incident that happened on a university VAX running Unix. In those days, everyone was using vt100 terminals, and the disk drives weren’t all that quick. He was working on his own terminal when without warning, he got this error when trying to run a common command (e.g.
ls)$ ls -l sh: ls: command not foundSo he went on over to the system admin’s office, where he found the sysadmin and his assistant, staring at their terminal in frozen horror. Their screen had something like:
# rm -rf / tmp/*.log ^C^C^C^C^C^C^C^C^C^C # ls -l sh: ls: command not found # stat /bin/ls sh: stat: command not foundA few seconds after hitting return, and the
rmcommand not finishing immediately, he realised about the errant space, and then madly hammered Ctrl-C to try to stop it. It turns out that the disk was slow enough that not everything was lost, and by careful use of the commands that hadn’t been deleted, managed to copy the executables off another server without having to reinstall the OS.Oh no, that’s horrifying.
I totally pulled a LTT and removed my kernel. >_< There was a “real time” kernel listed in apt, and I installed it because I was curious if it would reduce lock latency for a project I was working on. (I wasn’t trying to solve a problem, just curious) It didn’t and I figured it was probably a bad idea to leave it installed. So I did an apt remove, and the rest went something like this.
Apt: Are you sure you want to remove the your kernel? Y/N
Me: Oh jeez… I don’t want to do that.
Motor Memory: Y <return>
Apt: Are you really really sure? Your computer will not boot if you do this. Y/N
Me: Oh, crap! That’s not what I meant to do. Definitely not!
Motor Memory: Y <return>
Me: No! Why would my brain betray me!?
Fortunately this was on a PopOS machine, so I booted into the recovery partition. Even if fixing it only took a minute, I still felt very very dumb. >_<
Man, that’s a really dumb story that I find really relatable despite not having had any experiences like that. It feels like it’d be very in character for me though. Thanks for sharing, it helps me feel less silly in the various times where I’ve messed up (of which I am struggling to recall specific examples, but whatever brain part is responsible for embarrassment can remember, apparently)
The dumbest must have been when I went through the list of installed packages on Debian and removed everything named “python-…” since I don’t program in python.
This weirdly makes sense to me. Not long ago would have done the same.
-
have an nvidia GPU
-
have Fedora
-
download RPM package of drivers for Red Hat (after all, Fedora and Red Hat are… compatible, right?)
-
Everything goes fine
-
Six months later, upgrade to a new version of Fedora
-
oops, kernel panic at boot after the upgrade, and no video to troubleshoot after UEFI boot
-
figure out how to boot into a recovery partition from UEFI
-
figure out how to enable a serial console over a USB device
-
figure out how to connect to the serial console from another computer using another USB device
-
figure out what the kernel panic is from (not the upgrade, but the driver which wasn’t upgraded)
-
figure out how to uninstall the incorrectly installed driver
-
figure out how to install the correct driver
That was a fun three week OS upgrade.
I have a super-n00b question, and I apologize in advance, but, uh…yeah, what is a serial console?
You attach a secondary computer via serial (COM port) with your primary computer and then you can open a console on that one. You can access the primary computer as if you would be sitting in front of it.
You probably have to explain what Serial actually is.
I mean serial is just a port that runs in serial. You send something and you receive something afterwards, after you’ve received you can send again…
Not all people know that, to be fair.
True. It’s not quite common nowadays unless you work in administration or are an enthusiast.
https://en.wikipedia.org/wiki/Serial_port
Because there’s going to be kids around here who have never seen this port (other than maybe on a Point Of Sale (POS) system?)
Lol, I appreciate it, but I’m actually old enough to remember those! I know what a serial port is; I just didn’t know what a serial console was or if it was related. Haha.
Tl;dr: Stick in a USB cable and the other side gets your console.
https://en.wikipedia.org/wiki/Serial_console
tl;dr:
Serial ports are (for example) commonly RS-232, although other types of ports exist. Imagine it to be a very slow Ethernet device. Because it’s so slow (and the technology predates Ethernet and also has different requirements), it’s usually attached directly to a device instead of to a network. But you could connect a modem to it and it becomes connected to a network device.
It could also be connected to a system console device. These are commonly called terminals. Such devices are often monochrome (especially older ones) because a serial connection is often bandwidth limited (eg, measured in kilobits per second instead of megabits or gigabits). Since it’s so slow, it’s not practical for video, so it’s generally just text-only.
Note that your GPU might also output a system console but rendered on your display at very high resolution and with graphics-drawing capabilities. So a system console would be any console that connects to the system.
What is a console? Well, Wikipedia presents several valid articles and the common theme as far as computers go is that a “console” is typically something that a human and a computer use to interact with each other.
For serial consoles, you might find device files for them at
/dev/tty*. But for general serial devices, it could be any of several different types of device files.Wikipedia’s article on
/devdevices has a pretty decent listing of what kinds of devices you might find and several of them might be classified as a serial port. Any serial port might be connected to a serial console.So in my case, a serial console is:
- 2x USB-to-RS-232 (USB is a serial protocol and is basically “just” another (Universal) (and perhaps high speed) Serial port (Bus), so conversion is super cheap)
- 1x RS-232 null modem cable
That’s pretty much it in a nutshell. Then
- System 1 (the failing system) UEFI boots into repair system partition on a separately attached disk (eg, boot from CD or live USB) to get a local system console
- System 1 repair system mounts the failing system partition
- System 1 modifies failing system
grubconfiguration to enable a serial console on the attached USB-to-serial device file and saves changes, then unmounts failing system partition - Power off System 1
- Remove repair partition device
- Open terminal window on System 2 (recovery system)
- Connect System 2 terminal to the attached USB-to-serial device file using
screen(oh wow those were some old days) - Power on System 1
- System 1 boot enters grub recovery menu which allows fixing the system remotely
To be fair, a lot of that complexity could have been done by either reinstalling, or removing the hard drive and attaching it to another computer. But doing it this way allowed me to poke around and try different ways of solving the issue, rebooting, etc. It was a learning experience worth exploring.
It was years ago though and I think there was some complication with trying to understand what device file (or device number or something) needed to be to work on the correct serial device (there are often multiple)
Wait, that’s a tl;dr to you? o_O
I felt the same way so I scrolled down hoping for a shorter answer, but found yours instead and it made me laugh my ass off because how you wrote it really hit me, are you me? xD so I just wanted to say thanks for making my day even better!
Haha I’m really glad I could make another person laugh today! xD
I’ll quote my current boss’s boss’s boss when he asked a question of me:
@inetknght, can you please not write a book? I need a quick answer
Adding to what DmMacniel said, it’s a hardware interface, often accessed via a USB port (which after all, is the universal serial bus).
Christ you guys are making me feel old. I remember back in the day when a serial connection was made through an actual serial port. I know I have some serial cards around here somewhere. I have also used the tar command on an actual tape… Here’s a fun fact, if your tape drive (big reel to reel looking thing, not a cassette or other kind of ‘tape’) has an issue with rewinding, do not use your finger to manually spin the reel. Use a pencil. I finished reeling my tape back up once and realized I now had a blister on the end of my finger.
My motherboard which is only a few years old (2ish?) has serial port pin outs.
Why would you need a serial console? Live USB is a thing
-
If you count Android too, then this: I got my first Android phone when I was 10 or 11 and rooted it on the first day of having it. This was during a time when we were all still using ClockworkMod because TWRP didn’t exist yet, and I somehow ended up with a system without a kernel. Panic ensued, and I spent that entire night (like 10 hours) digging through xda in order to find a tutorial on how to get this damn phone to run again. Imagine having to tell your parents “I broke my phone I got yesterday.” I did get it working at like 6:30 AM. Fun times.
what’s the fun in modding if not the two hours where you think you’ve bricked everything and you’re scrambling through a 52 page post on XDA trying to find someone with your same problem
I did pretty much exactly this on a Galaxy S1 (i9000) that was old even when I got it, but my uncle who gave it to me said that to make it usable I needed to install Cyanogenmod.
I thought I fully bricked the phone trying and it actually sat dormant for years afterwards until I re-found the Odin backups I had taken, and was able to fully fix and restore it. Unfortunately by that time, nearly no ROM existed that was both up to date and a usable speed.
Oh man, Clockwork mod, that takes me back. Although I had my android phone for a while before I built up the courage to root it, in part due to stories like yours
dd if=/dev/zero of=/dev/sda status=progresshmm why is it so fast
OH
CTRL-C
and then a kernel panic yeah my fs was gone
Not me, and not Linux, but a school mate found the following bash snippet online
:(){ :|:& };:.Naturally, he tried it on the SunOS servers we had access to for schoolwork. He got his account suspended for the rest of the year.
I think most Linux distros are configured to kill fork bombs nowadays.
ntfsclone /dev/sdc /dev/sdb
/dev/sdb was a blank filesystem and /dev/sdc was my Windows filesystem.
It ran for less than a second and didn’t take me long to figure out what happened. That’s the story of how I stopped using Windows.
Damn that’s the equivalent of going cold turkey.
I don’t use windows for close to 20 years so I didn’t need ntfsclone so far but do I read correctly the man page that only the source is specified as positional parameter? If so, shouldn’t you have to write
nftsclone —overwrite /dev/sdc /dev/sdb? It still can be misleading (given that mv uses two positional parameters so mv -f source destination would have done what you wanted) but a bit less cryptic?
Yeah, sorry it was a long time ago (like 10+ years) but I checked and it would’ve been the --overwrite arg.
The manpage for the older ntfsclone command has it:
Clone NTFS on /dev/hda1 to /dev/hdc1: ntfsclone --overwrite /dev/hdc1 /dev/hda1
Moral of the story was to RTFM 😂
(RTFM = Read the Fucking Manual)
Adding this because I only learned this acronym just last week, and wish to share the knowledge with anyone else like me)
I installed
timeshiftto have a way to create restore points just in case I mess something up while fiddling with my Archbox.I used it for a while before I decided to remove it. After that, I realized it didn’t remove the “restore points” (I didn’t fully understand how it worked) and thought it would be good idea to
rm -rf /run/timeshift.My whole
/homewas smited (it uses symlinks to create these “restore points”). Before I realized, it removed gigabytes of data.Lesson learned: always understand how something works and always be careful when using rm -rf.
Best advice when using rm -rf
Don’t.Nice.
thanks
Due to some poorly placed quotes, I managed to create a subdirectory named
~in my home folder. You can imagine what happened next. Luckily, I had just gotten my backup system up and running the day before, so nothing was lost.How I lost a Postgres database:
- Installed Postgres container without configuring a volume
- Made a mental note that I need to configure a volume
- After a few days of usage, restarted the container to configure the volume
- …
- Acceptance
Well, that’s a dumb Docker thing, not necessarily a dumb Linux mistake. You could’ve made the same mistake on Windows or MacOS when running Docker.
Technically, containers always run in Linux. (Even on windows/OS X; on those platforms docker runs a lightweight Linux VM that then runs your containers.)
And I wasn’t even using Docker.
And then accidentally copy/pasting the failed attempt code snippets of the OP describing the situation.
I’ve regularly found a solution to my problem on SO, only to discover that I need to figure out how to break my system exactly the same way the asker did before the fix will work.
aptsomething that ended up removingsudo. No more admin rights.- used
rsyncto backup pretty much everything in / , with remove source option… findwith-deleteoption miss positioned. It deleted stuff before finding matching patternchown/chmodon/binand/or/usr/bin- Removed everything in
/etc
The first one can be fixed by using
suNot if root account is disabled. Which is by default on Ubuntu and Debian . You’d need
sudo su -but well… No sudo left you know.Damn
On the first point: isn’t it possible to just go su and reinstall sudo?
Or does it not work with disabled root?
It doesn’t work with root disabled.
The way to fix this is to boot in bash recovery where you land a root shell. From there you can hopefully
apt install sudoif deb file is still in cache. If not, you have to make network function without systemd forapt installto work. Or, you can get sudo deb file and all missing dependencies from usb stick andapt installthem from fs. Or just enable root, give it a password and reboot so you cansu -andapt install sudoThanks!

















