Grep duplicate JSON keys

If you have large JSON files with application settings in, you need to be sure that the settings only appear once. Not a problem until you get to the stage of very large files, being edited by all sorts of people manually.

"setting_1" : "some value",
"setting_2" : "another value",
"setting_1" : "different again"

Run a script to check for duplicate key names:

grep -Po '"[a-z_0-9]+"[ ]*:' <filename> | uniq -d

The above will output the duplicated setting(s) if any, to the console. Tested on Ubuntu 12.04

Backup all SVN repos

If you need to backup your SVN repositories, then you can use this bash script to do so:

DATE=`date +"%Y-%m-%d"`
mkdir -p $BACKUP_DIR
for dir in `ls /var/svn/`; do
    svnadmin dump $RES_DIR | gzip > "${BACKUP_DIR}/${dir}.dump.gz";

This basically dumps every revision from each repository in your server, and gzip’s them and then puts them in /home/user/svn/backup under the current date

Saving username and password OpenVPN config

I use the openvpn command line to connect to a certain VPN site, and I get annoyed at having to find the password everytime

So, in the .ovpn file, there is a setting:


So, I create a file (lets call it new-file.txt) in the same folder as the config, and it only contains 2 lines, first line username, second password

Change the auth-user-pass line to be

auth-user-pass new-file.txt

And now it connects without prompting me for password. Super.

Obviously this is a slight security issue, but this is on a pretty secure VM on a secure machine. So its quite safe. Just think about it if you are implementing the same workaround.

Log in to SSH with no password

So you want to have your password remembered by a remote SSH connection?

Easy, following these steps

On your local machine, create a new SSH key:


It will ask you for a file location and a keyphrase, but I always just hit enter three times.

This will create a file in the default folder (or whichever folder you chose in the first step)


The contents of this file want to go into a file on the remote server: If the file doesn’t exist, you’ll have to create it.

/root/.ssh/authorized_keys, or

That’s it. There are some cases where you have to change the permissions of the authorized_keys file but I’ve not had to do that. I think the suggested chmods are 640 and 700, but don’t hold me to that

From now on, you won’t be prompted for your password when SSH’ing in from the local machine. In order to revoke access, delete the right line from the authorized_keys file.

Finding files in Linux/Mac Terminal

Finding files is a little different to MS DOS!

find <startlocation> <type> <filename> <exec>

The exec is a simple way of performing functions on the results. If you leave it off, the find doesn’t really do anything.

So, the simplest thing to do is find a file, so the exec becomes the output:

find / -name wkhtmltopdf* -exec ls -ls {} \;

That will find all files starting with wkhtmltopdf starting from / and the function to run on it is ls -ls {} \;

The {} will be replaced with the search result, so, in essence, ls -ls filename; will show you the details about the file which is what I want, as I am looking for all copies of the wkhtmltopdf toolkit on the server.

You can obviously change the exec to delete the files, or copy/move them, or interrogate further…

New install of Centos – No network

If you have just set up a new Centos server and found that the network isn’t functioning, and you have no IP address for it then fear not, its a ‘secure’ default for Centos to not enable networking until you are ready to do so.

cd /etc/sysconfig/network-scripts/ 
sed -i -e 's@^ONBOOT="no@ONBOOT="yes@' ifcfg-eth0

Then reboot.

If this doesn’t help, then double check the filename in that folder. You may have ifcfg-lo instead.

More help here: http://wiki.centos.org/FAQ/CentOS6#head-b67e85d98f0e9f1b599358105c551632c6ff7c90

Compress files and folders in tar (Linux)

GZIP and TAR are two different utilities bound by the same cause. Compressing files and making file catalogs.

To compress a file, simply gzip it:

$> gzip database_dump.sql

That will compress the database_dump.sql file, and rename it to database_dump.sql.gz. Easy, and to uncompress it:

$> gunzip database_dump.sql.gz

That will restore the uncompressed version and rename to file back to .sql

But, what if you want to do more than one, or a folder. GZIP is not the tool, however it is used.

You need to TAR the files (which creates one file with the specified files in) and then gzip it:

$> tar -cvzpf compressed_file.tgz foldername

That will compress all files in the foldername folder into a file specified, and it will GZIP it for you.

If you want to just compress a few files, then instead of using the foldername have a list of the files separated by space

$> tar -cvzpf compressed_files.tgz file1 file2 file3

Easy, so to unTar them:

$> tar -zxvf compressed_file.tgz

The above will extract the files to the current folder, if you want them somewhere else:

$> tar -C /foldername -zxvf compressed_file.tgz

The above will extract the files to the /foldername folder.

Hope that helps a little.


Where is my.cnf on Mac OSX?

On Mountain Lion, I found that the my.cnf file is missing. This is because Mac uses the default MySQL settings and therefore doesn’t require a config file.

If you want to change the default settings, you’ll need to create a my.cnf in /etc/:

You’ll need to sudo into Terminal, and copy one of the default configs from the MySQL folder to the /etc/ folder:

sudo cp /usr/local/mysql/support.files/my-huge.cnf /etc/my.cnf

Then just use Nano or whatever to update the file, then restart MySQL and you should be good to go with your new settings.