Breck's Code Blog

Find all scroll files not matching a pattern:

grep -rL "endSnippet" *.scroll



Edit

My Sublime Settings

June 21, 2022

I'm going to start sticking my Sublime Text settings and plugins here.

"goto_anything_exclude_gitignore": true,

This setting is new devBuilds in ST4 (Build 4130 released on March 23, 2022). Set it to true and any file that matches your .gitignore patterns will not show up in ST's autocomplete, navigation, et cetera. Found this one on StackOverflow.




Edit

Git Clone over SSH

July 8, 2021

Using Git with your own servers—without a hub like GitHub or GitLab—is easy. This is the straightforward, quick and dirty way with no new concepts to learn. There are more advanced ways (like -bare), but I find 4 times out of 5 I just need a quick way to sync repos between machines.

Cloning from your server to your local machine

git clone ssh://breck@example.com/home/breck/project

Pushing to your server from your local machine

The one annoying thing here is your push will be rejected if your are pushing the branch that is currently checked out on the server. You can quickly check out a temporary branch, push main, and then check main back out on the server.

git push



Edit

Unpause

June 9, 2021

After a decade buried in the digital attic, I recently uncovered this old blog and upgraded the content to use Scroll. It was a fun restoration project and interesting to me personally to see what types of technologies I was using a decade ago. It was fun to read that I was still uncertain if I would switch to a Mac. IIRC that Macbook Air would last me until 2016! It was followed by another Air, and then around the end of 2020 I upgraded to the current crop of machines, typing this on an M1 MacBook Pro.

Anyway, I also just remembered how nice and convenient it was to post these small snippets I need in my day-to-day coding work on a fast, ad free site. So decided to restart this blog.

Why not just start a fresh one? That might be a better idea. For now I plan to reuse this one. Perhaps split them later, if that makes sense.

If I do keep with it, this post is just to acknowledge that that's no bug, there was a 3,444 day gap between posts.




Edit

Git Ignore Global

January 4, 2012

Create a global .gitignore file to ignore common pains:

vim ~/.gitignore_global .DS_Store* Icon? Thumbs.db git config --global core.excludesfile ~/.gitignore_global

Thanks to github




Edit

Ever want to non interactively execute commands on a remote server?

ssh breckyunits.com 'df -h'



Edit

Sometimes I want to rearrange a directory. An easy way to do it is to use Cinch and have 2 windows side by side. It's a pain to open 2 windows to the same directory though, until I found this little script




Edit

Find big files in Linux

October 6, 2011

Edit: found a better way, use ncdu https://dev.yorhel.nl/ncdu

How to find files larger than a certain size:

find /etc -size +100k

Disk free space:

du



Edit

ack and ackmate

August 26, 2011

So one common problem I have is writing code at a macro level---refactoring things in multiple files. Notepad++ used to make this easy with it's fast and powerful find/replace in folder feature. Textmate's built in find/replace in project is painfully slow.

Grep & Sed don't cut it either. I've been using some custom written perl scripts for my needs. But today I discovered ack & AckMate. Problem solved. Really tight integration with TextMate (though the default 4 key shortcut is terrible--need to change that).

This is probably something I would have learned faster had I been doing more pair programming with senior devs.




Edit

nohup and disown

August 22, 2011

Just read a good guide on how to keep commands running when you exit ssh:

Disown, nohup : Bash Commands.

Basically:

nohup ./mycommand &

Adding the ampersand to the end of a command in Nix executes the command in the background and gives you back your shell. However, if you quit your terminal the process will be killed. The nohup command takes care of that.

What is nohup? It's another obfuscated Nix low level utility that is an abbreviation for "No Hangup Signal", in other words: if the terminal sends a hangup signal, normally that kills all running processes, but this says ignore that.

Wikipedia article on nohup




Edit

Just read an excellent guide about how to create self-signed certificates.

Here's what it boils down to:

# The cool thing about SSL is it's just 2 text files. A public key and a private key. # They have a "pem" and a "crt" extension, but they are just plain text. Neat. # Okay, first step is to create a private key "pem" file. # every time you run this the random output pem file will be different: openssl genrsa -des3 -out private_key.pem 1024 # (enter passphrase) # Next, generate a "signing request". openssl req -new -key private_key.pem -out private_key_certificate_signing_request.csr # (for common name enter the full domain name: # (the extra options aren't necessary--just hit enter) # Next remove the password from the private_key. Otherwise you'd have to enter # that password at all sorts of inconvenient times.....be careful because # if someone gets this unencrypted ket, you'll need to get a new cert..so make your pem readable # only by root! cp private_key.pem private_key.pem.original openssl rsa -in private_key.pem.original -out private_key.pem # now you can generate your public self signed certificate: openssl x509 -req -days 365 -in private_key_certificate_signing_request.csr -signkey private_key.pem -out self_signed_public_certificate.crt # now copy the private key (pem file) and the public key ( crt file ) to your web server. mkdir /etc/httpd/certs cp self_signed_public_certificate.cert /etc/httpd/certs cp private_key.pem /etc/httpd/certs # you can delete the CRT file. you don't need that crap anymore. you got the cert/public key and the pem/private key. thats all you need rm private_key_certificate_signing_request.csr # if you don't have mod_ssl installed you need to install it. on fedora: yum install mod_ssl #now edit the apache config to let apache with mod_ssl know where the keys are: SSLEngine on SSLCertificateFile /etc/httpd/certs/self_signed_public_certificate.crt SSLCertificateKeyFile /etc/httpd/certs/private_key.pem



Edit

New shortcuts

August 14, 2011

Gmail:

gi - go to inbox c - compose j/k - go up and down, enter go to a page. gt - go to sent

Mac:

ctrl+a - home/start of line ctrl+e - end



Edit

EC2 Command Line Tools

August 13, 2011

Step 1. Download them. http://aws.amazon.com/developertools/351

Step 2. Set Java path in your .bash_profile:

export JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Home/

Step 3. test Java path is correct:

$JAVA_HOME/bin/java -version

Step 4. set ec2 path in your .bash_profile:

export EC2_HOME=/Users/breck/Downloads/ec2-api-tools-1.4.4.1 export PATH=$PATH:$EC2_HOME/bin



Edit

zip a directory in linux

August 9, 2011
zip -r backup mydir/ zip -r backup .



Edit

Recursive Sed

August 5, 2011

As far as I can tell, sed doesn't have a recursive mode. Here's a little script that uses perl to simulate a recursive sed:

#!/bin/bash perl -e "s/$1/$2/g;" -pi $(find . -type f)

I put this in a file named ~/rsed

Then in my .bash_profile:

alias rsed="~/rsed"

Now I can use it like so:

rsed old new



Edit

netstat

August 1, 2011

Every wonder what's listening on what ports?

netstat -lp



Edit

Bash History

July 17, 2011

Understanding and mastering bash history can:

  • 1. Save you a lot of time, keystrokes, and errors.
  • 2. Allow you to easily create shell scripts for common tasks (such as setting up a server).

Here's a terrific guide on using Bash History effectively.




Edit

Amazing SSH Tricks

July 16, 2011

Just read an article that contains some amazing SSH tricks.




Edit

I just came across a neat command line tool, ngrep, via this site.

Sample usage:

ngrep -q -W byline "^(GET|POST) .*"



Edit

Show memory usage:

free -m

Add this to your bash_profile:

alias free="free -m" top

Show all processes

ps -a

Disk free space:

df



Edit
  • Download the ISO
  • Convert iso to img on terminal:
hdiutil convert -format UDRW -o ~/path/to/new.img ~/path/to/source.iso
  • Open Disk Utility. Right click the drive and click Information, then note the drive path (disk1 or disk2 for example)
  • Right click the Disk and click Unmount
  • Back in terminal:
sudo su root dd if=/path/to/new.img of=/dev/rDISKNUMBER bs=1m

Where DISKNUMBER = disk1 or disk2 for example. Add that leading "r".

Back in Disk Utility, unmount/eject the disk.

Done!




Edit

If you hold the "option" key while using the Mac terminal you can use your mouse and click on where you want the cursor to go. This comes in handy if you mistype a command and don't want to retype the whole thing. It also kind of negates the need for a "home" key on the terminal.




Edit

I want my HAML files to have PHP(or Ruby) syntax highlighting.

Settings > Style Configurator

Then click php and under "user ext" (in the bottom left) add haml




Edit

Configuring Apache can be a pain. I'm going to try and collect links to great resources. The official docs are good, but they really need to add commenting so people can expand upon the resources available.




Edit

If you are installing Ubuntu from USB and are getting stuck at the "SELINUX 3.82" screen, you'll need to edit the selinux/selinux.cnf file on the USB stick and comment out the "vesamenu.c32" line, and remove the "ui" from the last line.




Edit

My Programming Setup

May 30, 2011

Here's my setup. This is kind of for my reference but you can also check it out if you're interested.

  • Macbook Air. 4GB. 256GB HD.
  • VMWare Fusion almost solely to run Notepad++
  • In Fusion I removed the Apple+W shortcut they map by default. Also, I map the Apple Key to the windows Ctrl key and the Mac control key to the Windows Key.
  • I configure Notepad++ to use 2 spaces instead of tabs.
  • I run Fusion in Unity mode and have the Windows taskbar at the bottom of the screen.
  • I make the Mac tray pretty small at the bottom of the screen and set it to disappear.
  • I run Cinch on my Mac to get the next drag and drop for side by side comparison function that comes with Windows 7.
  • I run Dropbox on the Mac. I share a folder using Fusion to have access to it in Windows.
  • I disable network access on the Windows virtual machine so I don't have to worry about viruses etc on that guy.
  • I enabled a firmware password on my macbook.
  • I add a lot of things to my bash_profile on the mac and pretty much always have at least 1 terminal window open.
  • I use textmate on the mac occasionally but try to do all my work in Notepad++.
  • I run MAMP on the Mac and WAMP on the WinVM.
  • I bump up the font size on a lot of things like terminal etc.
  • I go to "Keyboard" on the Mac and enable function keys. That way F1 works like you'd expect and you have to hit Fn+F1 to adjust the screen brightness.



Edit

I downloaded the Verisign Master .com TLD zone file which weighs in at a hefty 1.7 GB. What happens when I open it with different programs on my Windows 7 Samsung netbook?

With Windows Notepad: You quickly get a "File is too large. Open with a different text editor." message. Graceful FAIL

With Microsoft Wordpad: Immediately got a "Not Responding". On the status bar it says "3% complete" but 3 minutes later it hasn't changed. I'm going to go ahead and CtrlAltDelete this one. Disgraceful FAIL.

With Notepad++: "File is too big" error. Graceful FAIL.

VIM in Cygwin: It appears frozen.

I will continue this post at a later date.




Edit

Just create a new A Record with * as the host and your IP in the points to field.




Edit

Sometimes when abroad it becomes necessary to route your internet traffic through a proxy.

Here's a great article that shows how.

I also recommend the QuickProxy Firefox addon which lets you quickly toggle the proxy on and off.




Edit

I've been on the road for a few months programming on a Windows 7 Samsung netbook. (I didn't want to bring the Macbook with me and risk breaking it or worse).

The one thing I love about Windows is Notepad++. Sorry Textmate, you just can't compete.

Here are some things I'm using a lot more now in Notepad++:

  • Bookmarks. Hit Ctrl+F2 to set a bookmark on any line of code and then hit F2 to jump between bookmarks. Shift+F2 goes back a bookmark.
  • SourceCookifier. This is a neat plugin that lets you jump to the definition of a function by hitting Ctrl+Shift+Enter. This is one feature from IDEs that I missed. It also provides a nice sidebar with a breakdown of your code(functions, vars, etc.).
  • FileSwitcher. A plugin that lets you hit Ctrl+Shift+o and then just type the name of the file you want to work on. Much quicker way to switch files than clicking or ctrl tabbing.
  • Goto. Ctrl+G lets you jump to a line number quick. I've used this in the past but am using it more often now.
  • Column Editor. Pretty handy sometimes.
  • Automation PHP Plugin. F9 to run your script. Great for quickly testing small sections of code.



Edit

RVM Quick Reference

December 12, 2010

Return to the Mac OS X default Ruby install:

rvm system

Switch to Ruby 1.9.2:

rvm 1.9.2

Create a new gemset:

rvm gemset create rails3

Switch to newly created gemset:

rvm 1.9.2@rails3

Install rails 3 on new gemset:

gem install rails -v 3.0.3

Return to default ruby install:

rvm system

Install a new version of Ruby:

rvm install 1.9.2



Edit

Say you are running a web server locally and want to show it to someone else remotely.

You have a few ways to do this:

  • 1. Provide your local IP address to the person.
  • 2. Setup DNS to point a domain to your local IP.

But what if your local computer is behind a NAT and you don't want to configure port forwarding on your router?

One option if you have a publicly facing server already is to create a reverse SSH tunnel with that server. This allows you to forward a port on that server to a port on your local server.

For instance, on my Mac OS X machine I open terminal and type this command:

ssh -nNT -R 3333:localhost:3333 username@yourdomain.com

This will connect to my server (yourdomain.com), and forward traffic from port 3333 on that server to my local machine. Now I can tell someone to go to yourdomain.com:3333 and that would be the equivalent of them going to localhost:3333.

Notes:

Make sure to add a rule to your IPTables config to allow inbound traffic to port 3333.




Edit

Here are a few things I really miss from Notepad++:

  • When you doubleclick a word in Notepad++, all occurrences of that word are highlighted in green.
  • Notepad++ has great autocompletion. Once you type a word in a document, that word is added to the autocompletion list. Autocompletion drops down inline too which is great.

If anyone nows how to replicate these things in Textmate please let me know! Thanks.




Edit

Let's say you've got a file that's checked in like environment.rb that you want to edit for your local machine only and don't want to push the changes to your repository. It's easy, just do:

git update-index --assume-unchanged filepath

And to reverse:

git update-index --no-assume-unchanged filepath

Learn more here.




Edit

Adjusting to a Mac

October 2, 2010

For the third time in my life I've gotten a Macbook. The previous two times I returned them. I'm hoping the third time is a charm.

After a week I'm finally starting to get used to it. I'm starting to really customize it and that seems to make all the difference. I've removed nearly everything from the dock, added a bunch of things to my bash profile, and a friend at work gave me a crash course in Textmate.

I've downloaded a program called Sizeup, which not only emulates the awesome "Snap" feature of Windows 7, but in fact is even better. It gives you the left/right 50% snap feature just like in Win7, but also lets you snap things into top/bottom and even snap things into quadrants. Pretty nice software.

Firefox is my favorite of the 3 browsers on the Mac. I loved Chrome on Windows because it was so fast, but this Air is so blazing fast that running Firefox or Chrome on Mac makes little difference.

I've added the Secret Preferences menu and turned "show all hidden files and folders" on. I hate not seeing everything in a folder.

I created an alias "dsdelete" to quickly recursively delete DS_Store hidden files (helpful when zipping folders for sharing).

I learned the "open ." command and changed the terminal color scheme.

I installed RubyMine and Web Developer Toolbar and Tamper Data and Charles and Dropbox. I'm still trying out VMWare Fusion and haven't decided if I want it yet.

I've slowly gotten used to the keyboard. I tried spaces but have turned it back off for now. Expose I hardly use yet.

All in all, I'm really enjoying the speed, instant on, the light weight of the computer, Ruby on a Mac (doing Rails dev on Windows is a pain), having a real terminal(cygwin is great, but native is better), and the fact that 50% of my CPU isn't dedicated to fighting viruses. I'm still developing slower on this machine but hoping that my speed will improve with time.

I'd say the odds at this point are 60-40 in favor of me sticking with it this time.




Edit
find . -type f -name ".DS_Store" -exec rm {} \;



Edit

Rubyful Soup

August 14, 2010

I was a big fan of BeautifulSoup in Python and was excited to see a Ruby port. Sadly it's no longer being maintained.

To get v1.0.4 working with Ruby 1.9.2, I had to make the following changes to the rubyful_soup.rb file in my gems directory:

  • Line 19 - comment out 'require 'jcode''
  • Line 233 - remove the colon



Edit

Windows 7 Hosts File

August 14, 2010

The file is located at:

windows/system32/drivers/etc/host

Mac hosts file:

/private/etc/hosts



Edit

VIM and Rails

August 13, 2010

Check out this guide.

filetype on " Automatically detect file types. set nocompatible " We don't want vi compatibility. " Add recently accessed projects menu (project plugin) set viminfo^=! " Minibuffer Explorer Settings let g:miniBufExplMapWindowNavVim = 1 let g:miniBufExplMapWindowNavArrows = 1 let g:miniBufExplMapCTabSwitchBufs = 1 let g:miniBufExplModSelTarget = 1 " alt+n or alt+p to navigate between entries in QuickFix map <silent> <m-p> :cp <cr> map <silent> <m-n> :cn <cr> " Change which file opens after executing :Rails command let g:rails_default_file='config/database.yml' syntax enable set ruler " Ruler on set nu " Line numbers on set ruler " Ruler on set nu " Line numbers on set nowrap " Line wrapping offset nowrap " Line wrapping off set timeoutlen=250 " Time to wait after ESC (default causes an annoying set ts=2 " Tabs are 2 spaces set laststatus=2 " Always show status line.



Edit

PHPUnit Filter Option

August 10, 2010

Say you are running PHPUnit from the command line and want to run just a specific test as opposed to a whole class of tests.

For example, say you have a tests.php file that contains a FileWriterTestClass with a test method named testFileWriter that you want to run.

Do this:

phpunit --filter testFileWriter FileWriter_Test_Class tests.php



Edit

From your rails app's root:

cat tmp/pids/server.pid kill -9 {pid}



Edit

Add this to your /etc/defaults/etc/ssh_config file to prevent SSH servers from disconnecting you after inactivity:

ServerAliveInterval 60



Edit

Augmented Gitignore

August 4, 2010
.bundle db/*.sqlite3* log/*.log *.log tmp/**/* tmp/* doc/api doc/app *.swp *~ .DS_Store



Edit

Windows Flush DNS

June 30, 2010

Open cmd.exe. Run:

ipconfig /flushdns



Edit

If you have a massive text file you want to add a line to without opening the file, here's a quick solution:

sed -e '1i\text_to_insert' -i file.txt



Edit

wget don't save file

March 27, 2010
wget -O /dev/null http://address.com



Edit

Dump MySQL schema only

March 10, 2010

I found this command to dump just the schema.

mysqldump -u root -pmypassword test_database --no-data=true --add-drop-table=false > test_dump.sql

You can then load the file easily.




Edit

gca alias for faster git

March 10, 2010

One of my most frequent commands is:

git commit -am "message"

I made a simple alias to save keystrokes:

alias gca="git commit -am "



Edit

ls color scheme

March 10, 2010

Found this helpful color scheme for ls. toss in your bash_profile.

alias ls="ls --color=auto" LS_COLORS="di=31;1:ln=36;1:ex=31;1:*~=31;1:*.html=31;1:*.shtml=37;1" export LS_COLORS



Edit

Recursive ls

March 10, 2010

Add the -R option to ls to recursively list the contents of a directroy.




Edit

vim macros

March 9, 2010

Let's say you want to add comments to your php functions following the conventional format:

/** * Returns the sum of 2 variables. * * @param int $a * @param int $b * @return int $sum */ function sum($a, $b) { $sum = $a + $b; return $sum; }

You can record a macro to help speed up the commenting process.

Just type q followed by the letter you want to assign the macro to (in this case let's use c), type your keystrokes, then hit q again to stop recording. Now hit c to replay your macro.

You can overwrite the macro by repeating the above steps.

Bonus tip:

Bookmarks.Use m + a letter to create a bookmark in vim. Then use the backtick plus the letter to return to the location.

Keyword completion. Use Ctrl+N to complete a word.

Notes




Edit

To get the actual directory, run:

/bin/pwd



Edit

On linux, filenames and directories are case sensitive. file and File are different files.

On Windows, filenames are not case sensitive. file and File are the same file.

MySQL saves a database to disk. A "database" is saved in a directory, and each table is saved as a file in that folder.

Thus, say you transfer a database from Linux to Windows that has a table named "userBookmarks", this might cause a problem for you because MySQL on Windows will likely interpret it as "userbookmarks".

There's a setting in your my.cnf file called "lowercasetable_names".

The options are 0, 1, and 2.

1 is the default, and you shouldn't need to change it.

If you change it to 0 on Windows, you might run into problems. If you change it to 2, it doesn't seem to do much.

So what's the solution? Try to avoid capital letters in table names (use _ instead of camel case). When you can't do that, write your code for the Linux database (ie "select from userBookmarks"), and your queries should translate fine on Windows since Windows will convert that to "select from userbookmarks".




Edit

The key things you need to know:

ANSI (aka ASCII) and UTF-8 (aka "UnicodeTransformation Format" or just "Unicode") are ways to encode text files into binary data.

I'm simplifying things but actually what I'm leaving out is not important. There are basically 2 character sets: ASCII or ANSI and UTF8 or Unicode. It's probably easiest to just call them ASCII and Unicode.

Unicode is the new guy, and is slowly replacing ASCII.

The main problem you'll run into with ASCII is line breaks. This isn't really an ASCII issue, it's a Windows issue but it often seems like it's an ASCII issue.

Windows does \r\n, whereas Unicode just does \n.

Here's an experiment you can do using Notepad++ and Notepad.

Open Notepad++ and start a new document.

Type in:

hello<br>world

Click "Edit", "EOL Completion" and click "Windows Format" (if it's gray, that means it's already on Windows format). Now cut and paste the text into Notepad. You'll see that it comes out on 2 lines. All good right?

Now go back to Notepad++ and click "Edit", "EOL Completion" and click "UNIX Format". Now cut and paste the text into Notepad. It's on 1 line, right?

Tada!

Now when you're having line problems just use this nifty feature in Notepad++.

Notes




Edit

I used to use Python, Ruby or even PHP to write backend programs that would automate things like server management tasks, development and editing tasks, deployment tasks, backup tasks, and so forth.

Then I learned what is basically the BASH programming language, a language very similar to Python/Ruby/PHP etc., but is perfect for writing command line programs.

Here's the core of what you need to know:

Use the .sh file extension

A BASH script should have the .sh extension, just as a Python script has the .py extension, a php script has the .php extension, and a Ruby script has the .rb extension.

Start your scripts with #!/bin/bash

#!/bin/bash

Here's your first bash script (name it hello.sh):

#!/bin/bash<br> echo "Hello World"

The echo command prints a string and also a new line at the end.

Running your script

Your script must be *executable*. To make it executable, you need to change the file permissions. This would make your hello.sh script executable:

chmod 777 hello.sh

To run your script, from the directory where your script is located, type this:

./hello.sh

The "./" substitutes the current directory. In other words, typing

./hello.sh

Is equivalent to typing the full path to the file such as this:

/home/breck/hello.sh

Command line Paramters

Command line parameters are accessed with $1, $2, etc. Create a new script called hello2.sh. Put this in it:

#!/bin/bash echo "Hello World" echo $1 echo $2

Now run your script like this:

./hello2.sh hi breck

It should print this:

Hello World hi breck

The hi got stored in $1, and the breck got stored in $2.

Functions

Functions are pretty simple and straightforward:

#!/bin/bash echo "Hello World" echo $1 echo $2 function hi_name { echo "hi" echo $1 } hi_name $3

Notice you don't need parenthesis and other junk. Name this file hello3.sh and run it with this:

./hello2.sh hi breck conor

It should print this:

Hello World hi breck hi conor

Notice what's going on with parameter passing and scope.

That's it for now. As you can see, BASH is a very clean, simple, yet powerful programming language. Get started with it slowly and gradually build up your skills with it.




Edit

I've been getting this error a lot in Cygwin on Windows 7:

Could not resolve hostname X: Non-recoverable failure in name resolution

It happens a lot when I run a deploy script or git push. Haven't found a solution yet, and am trying to track one down.

The problem is this bug is hard to reproduce. It happens only 5-10% of the time.

Edit #1:

Edit #2:

ssh: Could not resolve hostname X: Non-recoverable failure in name resolution

That's narrowed it down a bit but still no solution. Sadly because it only happens once or twice out of 10-20 times I run the "ssh" command, it's taking longer to pin down.




Edit

Plugins for Notepad++

February 6, 2010

Notepad++ is my go to editor.

The default plugins are great, but if you need more just visit the notepad++ plugins list.

To install a plugin:

  • 1. Download the zip file and unzip it.
  • 2. Copy the dll to your Program Files/Notepad++/Plugins folder.

The main plugin I've used so far is Explorer. I'll update this in the future with other ones.

That's it!




Edit

My post on deploying a website using rsync has gotten quite a few hits.

I thought I'd add some more tips:

  • use the "--delete" option if you want to delete files on the destination directory that no longer exist in the source directory.
  • add "$1" after your options like this:
rsync -rvz$1

Then you can easily do a DRY RUN. Just type:

./deploy.sh n

The n will get substituted for $1.

Troubleshooting

I've found a lot of times rsync appears to be copying the same files over and over.

There are a few things you can do to troubleshoot when rsync recopies the same files.

The options that I've found come in real handy are:

  • 1. The i option, which shows you why a file is being copied (all pluses means rsync thinks it doesn't exist in the destination; t means the times differ; s means the contents differ (?)).
  • 2. The --modify-window=1 option. This lets you ignore differences in modified timestamps. You can set 1 to a high number to ignore large differences.



Edit
cd -

To go back to the last directory you were in. Basically an "undo" for the cd command.




Edit

There are two ways to organize source code files for a project: flat or nested.

Here's a simple example of a flat structure(1 directory, 4 files):

📁myproject index.php theme.css helper_functions.php logo.png

Here's an example of a nested structure(4 directories, 4 files):

📁myproject index.php 📁styles theme.css 📁images logo.png 📁 helpers helper_functions.php

Both of these structures can accomplish the same thing. Which is better?

If you've worked with me on any project before, you'll know I prefer flatter structures. I try to resist adding directories as much as possible (you always have to add them eventually, but I like to fight it to minimize this effect).

I don't know why my gut tells me to minimize directories, so I thought I'd write about it and see if any rationale emerged.

Benefits of a Flat Structure

Benefit #1 - Quicker access to files

I primarily edit files using Notepad++ on Windows 7 or vim. If you keep all your files in 1 directory, you can open them faster ("Control+O", type a few letters on Win 7, or just "vim filename"). So, a flat structure gives you quicker access to files. I've never timed this before, but am pretty positive it works.

Benefit #2 - Avoid file path issues

Dealing with paths when working with a website is often the cause of careless bugs. If everything is in one directory, it makes it painless to include things on both the server side(include 'file.php';) and the client side ('src="script.hs"'). Once you start nesting directories, it's easy to make mistakes.

Benefit #3 - Easier to bring people up to speed

When I join an in-progress project or use a new library, I like to try and wrap my head around the whole thing. I can't do that if I have to dig into different directories to see all the files. Why hide 50% of the project in subdirectories?

Benefit #4 - Avoid overwriting issues

Occasionally you'll have a structure like this:

index.php 📁admin index.php

You might accidentally overwrite one index with the other, or edit one index when you think you're editing the other. Doesn't happen too often, but occasionally it can be annoying. By sticking everything in one directory, you don't run into that problem as much.

Benefits of a Nested Structure

Of course there are benefits to having multiple directories, otherwise this style wouldn't be so popular.

Benefit #1 - Looks more organized

It's nice and neat to have your files tucked away into nicely labeled directories.

Benefit #2 - Easier for teams

It can be easier to divide work between team members when things are in different directories. For instance, why should the designers need their images mixed in with backend files?

Benefit #3 - A directory with too many files becomes unwieldy

I'll admit, once you have more than some threshold of files in a directory, it can quickly become hard to manage. If the files are nearly all of the same type (say, php files), then it's not as bad. But once you have 20 php files, 3 javascript files, 15 images, 4 css files, 2 bash files, 3 readmes, and 45 text files, it might be time to split things up.

My theory on why directories are bad.

Less paths create an easier mental model.

If you walk into a messy room, it can be hard to create a mental model of the contents of that room. This is the equivalent to a file structure like the one below:

Bad

📁myproject a_image.png base_functions.php dog.js index.php mike.png names.csv new_logo.jpg test.php xmlfunctions.php zlib.js

It's a mess.

If you walk into a room where everything is put away(clothes in the dresser, books on a bookshelf, odds and ends in the drawers) it's easier and an improvement. A room organized like this looks like the file structure below:

Ok

📁myproject index.php 📁images a_image.png mike.png new_logo.jpg 📁php base_functions.php test.php xmlfunctions.php 📁scripts dog.js zlib.js 📁data names.csv

But now you've introduced the problem that to access nearly anything, you have to "open a drawer". You also can't see everything all at once.

Best

The optimal solution is to not put your stuff into drawers, but to:

  • 1. Remove as much stuff as possible.
  • 2. Put the remaining stuff out in the open in neats piles in separate parts of the room.
  • 3. If, and only if, you have a ton of the same type of thing, put those into their own drawer/cabinet.

If you apply this algorithm to a sample project, you might get something like a project I just made with PHPUno

📁dropdate 📁data 1 2 .. 578 index.php logo.png sprites.png style.css uno_controller.php uno_app.php uno_models.php uno_helpers.php

Search, don't sort

Gmail has an adage "search, don't sort". I think it applies here. Directories are a form of sorting, while autocomplete(which is integrated into practically everything nowadays) is a form of searching. By sticking everything into one directory, you enable search. Create multiple directories, and you disable search.

The Rule of Three

Here's a simple rule of thumb to help you organize your folder structure better:

  • No folder in your project should have more than 3 subfolders.
  • There should not be more than 3 levels in any project.

In other words, the directory "icon" should never have a subdirectory in it.

📁myproject 📁images 📁icons

This means that any project should have at most 9 subdirectories.




Edit

Unix date command

January 19, 2010

Display the current unix timestamp:

date +%s

Handy way to view it from the command line.




Edit
nullData = new Array( ); myChart.setDataArray(nullData,"null");

Add these 2 lines to your javascript.

Basically, if the first data set has only 1 data point, JSCharts will fire the alert. If it has 0 or 2+ data points, it won't. This fix creates an empty data series which disables this alert.




Edit

Change the owner of a file or folder:

chown newowner filename

Change the group of a file or folder:

chgrp newgroup filenameorfoldername



Edit

It's very simple thanks to pipes.

mysql -u username -p < sqlfile.sql

Make sure your sql file has a "use database" line.




Edit

In cygwin it can be a pain to change to a Windows directory like C:\Users\breck\Documents\My Dropbox.

I made a small bash script called wcd that makes it easier.

wcd

#!/bin/bash echo $1 newpath=$(echo $1 | sed 's|\\|/|g' | sed 's|:||g' | sed 's|^C|/cygdrive/c|') echo $newpath cd "$newpath"

The echos are just for debugging--I'm kind of new to bash scripting. Feel free to remove.

Make this file executable and add an alias into your bash_profile:

alias wcd="source ~/wcd"

Now you can just type(or actually copy/paste) something like this:

wcd "C:\Users\breck\Documents\My Dropbox"

...and that will take you to the Windows directory.

Notes

  1. wcd gist.



Edit

In Linux every thing is a file. This is great because it means the Windows clipboard is a file too!

Want to copy the current cygwin directory into your clipboard?

pwd > /dev/clipboard

Want to print the contents of the clipboard?

cat /dev/clipboard

Notes

  1. Fun with Cygwin's /dev/clipboard.



Edit

New tricks in Vim

December 29, 2009

I learned some new tricks with vim today.

4G - Move to line #4 mb - Mark the current spot as point b 'b - Go back to point b vim file1 file2 - open 2 files at once :e file - edit a different file :split file - split the current window and edit a different file (vsplit for vertical split) ctrl+w j/k - move down/up a window :hide - hide the current window :ls - show current buffers

Notes

  1. [Vim Tips and Tricks](http://www.cs.oberlin.edu/~kuperman/help/vim/windows.html)



Edit

In linux, standard input refers to the stream of bits that come from your keyboard.

Standard output refers to the stream of bits that appear on your screen(terminal).

You can change this around so that standard input comes from another source(say, a file) or so that standard output gets directed somewhere else (say, a file).

For example, say we wanted to email someone a file containing the contents of our home directory. We could do this:

ls ~ | vim -

This will pipe the output from the ls command to vim. You can then edit and save this file normally as you would in vim.

There's another type of pipe you could use for this example: greater than.

ls ~ > contents.txt

This will redirect the output to the contents.txt file.

You can also append output by using two greater than signs:

ls ~ >> some_file.txt

As you would expect, the less than sign can be used to direct a file as standard in.

Notes




Edit

Want to know how to run two or more linux commands sequentially? It's simple: use the semicolon ;.

For instance:

cd;ls

This will change the directory to your home directory, then list the contents. You can use as many semicolons as you wish.

Another option is to use && if you only want the next command to execute if the previous command executed successfully.

cd ~/myfiles; cp file.txt ~/backup/ && rm file.txt

This will only run the last command if the cp command executed successfully.

You can swap the && for || to run a command if the previous one fails.




Edit

Print the contents of a file:

cat filename.txt

Pipe the contents of the file to less, so you can view it one page at a time:

cat filename | less

In less, the 3 key commands are:

  • spacebar to go forward one page
  • b to go back one page
  • q to quit

You can also do regular expression searches like in vim: /regex (n to go to next match)

Append file1 to the end of file 2:

cat file1 >> file2

Combine two files into a third file:

cat file1 file2 > file3



Edit
What's the difference between a great programmer with great social skills and a great programmer who does not use version control?
Nothing--they both don't exist!
If *every single great programmer* uses version control, why do some beginning programmers avoid it?
Because version control is made to seem intimidating. But version control is actually very easy!

If you are new to version control, there are only 5 commands you need to memorize. Can you memorize 5 words? Of course you can.

So memorize these 5 words and you'll be practically an expert at version control:

init add status commit push

If you just start playing around with these 5 and only these 5 commands, you'll become a git master in no time.

Here's a simple practice session you can follow to start getting good.

What is git?

Git is a simple command line program like "wget" or "vim" that you install and use by typing commands. If you don't have git installed, try one of these commands:

yum install git-core apt-get install git-core sudo port install git-core

Let's say you're creating a new website for your mom and want to use version control to do it.

mkdir moms_website cd moms_website git init

This creates a git repository. Now type:

ls -a

Do you see the new ".git" directory? That's the git repository. It's basically a folder that stores the whole history of your project. Now, when you type a git command, it will do something with the files in that folder. That's all that's really going on. You never need to go into that folder manually, I was just explaining what git is doing.

Now, let's create a file and add it to your repository.

vim index.php Hello World :wq git status

This will show the presence of an untracked file, "index.php". Let's add this file.

git add index.php

You've now added the file to git, let's commit our changes.

git commit -m "first commit"

Now you've made your first commit.

The last command you'll need is push. It works like this:

git push

That will upload your repository to an online host like github so that other people can collaborate.

Create a github account and follow the instructions for creating a new repository to test out this final command.

That's it! Those are the 5 commands you'll use over and over again. Master those and slowly you'll start learning a few other helpful git commands.

Recap

git init git status git add *filename* git commit -m *"your message about what you changed and why"* git push



Edit

You can easily grab a file off a server from the command line using secure copy:

scp user@domain.com:/home/user/fileyouwant.txt fileyouwant.txt

This will download the file from your server to your local computer.




Edit

Clear screen in cygwin

December 20, 2009

Just hit ctrl+l




Edit

Use sed to find and replace

December 20, 2009

grep is to find what sed is to replace.

Say you noticed a few typos in a file. Say you typed imposible instead of impossible a few times in a file named "stuff.txt".

You can fix all occurrences using this one-liner:

sed 's/imposible/impossible/g' stuff.txt

Often it's more helpful to sed across multiple files.

sed -i 's/1.0.1/"1.0.1"/g' *.php

That's one command I used to add quotes around 1.0.1, which I forgot to do. Alas, it's not recursive.

Notes




Edit

Bash variables

December 18, 2009
$ FIVE=5 $ echo $FIVE 5 $ echo "The number five is $FIVE"

Variable scope

A shell script will launch a new shell that has a fresh scope.

$ FIVE = 5 $ vim myshellscript.sh #!/bin/bash echo $FIVE $ ./myshellscript.sh

Blank because the executed script doesn't affect the current shell. To make something affect the current shell, use the source command.

Export command.

Use the export command to change the scope of your variable to global for your shell.

To make a permanent global variable, add it to your .bash_profile like so:

MY_NAME=breck; export MY_NAME

Accept input.

Use the read command to accept input.

echo "What is 1+1?" read ANSWER echo "Your answer is $ANSWER"

Notes

  1. Bash ">tutorial 2. Another ">tutorial 3. Bash ">parameters




    Edit

I have used the same Gmail address as my main email for over 5 years. Since then:

  • I've made dozens of other emails that forward to this address.
  • I've signed up for hundreds of web apps and newsletters.

As a result, my Gmail account gets inundated with email. Here are the stats:

  • 16,054 unread emails in my inbox
  • 29,236 emails total in my inbox
  • ?? who knows how many total emails in my archive
  • 3.683 GB of email
  • Using 49% of my Gmail quota

My Gmail account had become a firehose.

This was bad for two reasons.

First, I started missing important emails. Not many, but a few per month. I had to write my own simple Bayes filter to find them.

Second, I couldn't do what most people do: use my inbox as a "todo" list. Most people immediately archive or delete an email when they're done with it. This is a great organizational trick. I wanted to do this again.

I considered starting a new email address and giving it out to important people, but that seemed like it would make my life more complex.

As it turns out, the solution was simple.

  • Started liberally using the "Spam" button. I started clicking Spam on pretty much any type of mass email that wasn't very interesting to me. Will be interesting to see how this plays out.
  • Starred all the recent messages that I still had to deal with.
  • Archived all 29,000+ of my gmail messages at once.
  • Went back to the starred folder and moved the important messages to inbox.

Notes

  1. Thanks to this site for telling me how to archive all my gmail messages.



Edit

The ls command lists all files and directories. What if you just wanted to list the directories?

ls -l | egrep '^d'

Add this to your .bash_profile and now you can just type "lsd" when you want list only directories. Who said lsd was a bad thing?

alias lsd="ls -l | egrep '^d'"

Notes

  1. Found this tip here.



Edit

Add these to your .bash_profile to speed up common tasks:

alias apr='sudo ./usr/sbin/apachectl restart' # restart apache alias www='cd /var/www/html/' # enter your root web directory alias hconf='vim /etc/httpd/conf/httpd.conf' # edit your apache config file alias eprof='vim ~/.bash_profile' # edit this file alias rprof='source ~/.bash_profile' # reload this file (after making edits) alias pconf='vim /etc/php.ini' # edit php.ini alias lsd="ls -l | egrep '^d'" # list only directories



Edit

git clone without history

December 11, 2009

If you're using a library from git in another project, you may want to download it without the history. Here's how:

git clone --depth 1 your_repo_url

http://git.or.cz/gitwiki/GitFaq#HowdoIdoaquickclonewithouthistoryrevisions.3F




Edit

Turn off bell in cygwin

December 11, 2009

The beep on tab completion in cygwin can be annoying. Here's how to turn it off:

vim ~/.inputrc (Then add or uncomment this line) set bell-style none

Save and restart your shell.




Edit

Create a new user in Linux

December 5, 2009

Creating a new user in Linux is dead simple.

useradd bob passwd bob

This adds a user "bob" and then prompts you to enter a password.




Edit

Symbolic links are simple and can save you time.

Here's all you need to know:

ln -s actual_file_or_directory new_symbolic_link_name

Let's do a real example from your home directory:

ln -s /var/www/html/yourwebsiteroot/yourblog blog

Now when you login to your server you can just type "cd blog" to access the "/var/www/html/yourwebsiteroot/yourblog" folder.

Here's one more example from your home directory:

ln -s /etc/httpd/conf/httpd.conf httpd.conf

Now you can just type "vim httpd.conf" and edit your httpd.conf file quicker.

You can do all kinds of neat things with symlinks, but the simplest uses are often the ones you'll use most. (One of the cooler uses of symlinks is for deploying different versions of websites).

Notes

Aliases are another great time saver.




Edit

Use aliases to save time

November 28, 2009

Using aliases in Linux can save you a lot of time. Here's an example:

alias www='cd /var/www/html/'

Instead of typing "cd /var/www/html/" to enter your web directory, now you can enter "www", a savings of at least 14 keystrokes each time you use it.

This one will switch to the usr/sbin directory, restart apache, then switch to your home directory:

alias apr='cd /usr/sbin/;sudo ./apachectl restart;cd'

You can also just do:

alias apr='/usr/sbin/apachectl restart'

But the earlier example shows how you can string multiple commands together.

Bonus tip

Place these aliases in your HOME/.bash_profile file and they will be available each time you login to your machine.

Notes

  • 1. "At least" 14 keystrokes, because the # of typos increases as the # of characters typed increases.



Edit

How I organize my Dropbox

November 27, 2009

Gmail encourages users to "search not sort". I do that. Email works great for search. For files though, sorting is often better.

I have well over 20GB of files synced onto my Dropbox. The rest of my computer is spotless(I use Windows 7 and do a fresh install every 30-60 days). But the Dropbox can get cluttered. I take a simple approach that is a hybrid of search and sort. Here's how to do it:

Put files into 2 sections:

  • Create a folder in your root Dropbox directory named "Search". Plop anything in here.
  • Keep your most common folders in the root Dropbox directory. Sort these ones.
  • Every few weeks, move any rarely used files and folders into the "Search" directory. Bring any commonly used files and folders back out to the root Dropbox directory.

End Result

Here's what the end result may look like:

-My Dropbox --Music --Photos --Blog --eBooks --Company Docs --Search

The first directories are the ones you access a lot and are well sorted and maintained. The last directory is a mess of random files and folders that you've collected. You can browse through it, but it's easier to just run a search on it when you need something.

This a hybrid of search and sort.

Notes

  • 1. All of my coding work is done in my cygwin home directory and then pushed to github. These are the only files that I don't store in Dropbox, but I push them frequently so don't worry much about losing them.



Edit

Say you want to get just the source code of a git repository without all the history(without the ".git" directory). Use this:

git clone --depth 1 git://github.com/breck7/brecksblog.git



Edit

Here are the programs I use everyday, every week, and every month.

What software do you use the most? What are your "must haves"?

My Machines

My primary development machine is a basic Compaq laptop with 4GB, 230GB, x2 Core Athlon, running Windows Vista/7.

My secondary machine is an Asus EEE netbook with XP which is great for working on the road, at a coffee shop, or on a plane.

Most of our servers are slices from slicehost running Fedora.

On all my development machines I run Cygwin. Programs I run in Cygwin are noted by *.

The following is a list of the programs I use on my development machines.

Daily

  • Cygwin. I prefer Windows+cygwin over Macs, Fedora or Ubuntu, or Windows plus running virtualized machines.
  • Notepad++. I do half my editing in Notepad++ and half in Vim. I love Notepad++'s speed, reliability, and autocomplete.
  • Dropbox. Completely changed how I work. Allows me to work from multiple computers seamlessly. Allows me to share files easily with clients, family, and friends. Reinstalling Windows (which I do a few times per year), is made easy because all my important files are kept on my dropbox. The fact that install and uninstall is so easy and doesn't require a reboot really impresses me.
  • WAMP. The "AMP" stack is what I develop for mostly. WAMP gives me an easy local testing environment to try things out before pushing to a server.
  • Chrome. I spend most of my web time in Chrome. It's lighting fast. It's also simple. Tabs are done incredibly well.
  • Firefox. Still need this guy. The extensions are great. Firebug and the Web Developer's Extension I use at least weekly, close to daily. It's also nice to have 2 browsers going if you want to log in to the same site with 2 different accounts.
  • Skype. I don't use a cellphone, believe it or not. My Google Voice number points to my Skype In number. Skype allows me to not have a cellphone, which I greatly appreciate. My sister can call me from the Middle East just as easily and for the same price that my neighbors can call me from down the street.
  • *Vim. The only Nix editor I know and love. Vim took me a while to learn, but is becoming more and more indispensable the better I get at "typing in Vim".
  • *Git. Easy to use version control that makes sense. Paired with github, becomes indispensable.
  • *Rsync. Great for deploying simple websites and for downloading backups.
  • *Grep. Great when you need to track down a word or line across multiple files.
  • *Find. Simple and easy for finding files. I also use find over Windows 7's search occasionally.
  • *Cron. Simple and essential.
  • *SSH. Of course.
  • *Ruby. I keep irb running in a window when I need to do quick calculations.

Weekly

  • Navicat. Great way to keep an eye on databases. I also like it for creating and editing schemas. Find it easier to visualize a schema than to code it out.
  • Filezilla. Occasionally when I just want to drag and drop some things from or to a server.
  • Photoshop. I use an old version of Photoshop which is still way to advanced for me. But it's fast and gets the job done.
  • 7zip. "Extract to here" is a great little feature that it adds to the right click menu.
  • Word & Excel 2007. Solid programs. Too bulky for my liking, but necessary in many cases.

Monthly

  • Visual Studio. When I get an urge to write a new desktop program.



Edit

cron in 2 steps

November 25, 2009

cron lets you schedule linux commands to execute on a regular basis.

For instance, say we want to run the command "wget http://yourdomain.com/backup.zip" each hour, that downloads a backup of one of your websites.

Step 1: Write the cron command

Here is the cron command you would need:

0 * * * * wget http://yourdomain.com/backup.zip

You can either memorize the crontab syntax, or just this simple Javascript crontab gui will easily generate the code for you.

Step 2: Append the command to your crontab file

There are 2 crontab files that get executed every minute. 1 of them is found in /etc or similar and is controlled by root only. In addition, each user can have a crontab.

To add your command to crontab type:

crontab -e

This will open vim, and you can then paste and save your crontab file.

Viola!

Notes

  1. If you are using cron in cygwin, follow these steps after installing the cron package with the cygwin installer to install cron as a service:
cygrunsrv -I cron -p /usr/sbin/cron -a D net start cron

If there are problems, check the cron.log file in your home directory.




Edit

Permission problems driving you crazy? Here's a common command to fix them:

chmod -R 755 directoryname

Change the file permissions of all directories and files in directoryname to 755.




Edit

Say you're looking for all files starting with the word "breck". Using the linux "find" command, it's simple:

find -name "breck*"

Will search the current directory and all subdirectories for files starting with the word "breck".




Edit

grep lets you search through multiple text files at once.

I recently removed a feature from a project and wanted to make sure I removed all references to it. With grep it was easy:

grep -r picture *

This searched all the files in my current directory as well as all the files in each subdirectory, and printed the filename and line where the word "picture" appeared.

Then I went in and changed the 2 files that needed modifying.

Had I wanted to just search the current directory (without the subdirectories) I could have done:

grep picture *.*

This would have searched through all the files in the current directory.

Had I wanted to search in just 1 specific file:

grep picture filename.txt

Had I wanted to do a case insensitive search:

grep -ir picture *

Had I wanted to exclude a directory and done a case insensitive search:

grep -ri "searchstring" --exclude="*\.svn*" *

(grep uses regular expressions)

Links




Edit

rsync is a nifty little tool that can deploy a website to a testing or production server.

Here's a one line command to deploy brecksblog:

rsync -arvuz /home/user/brecksblog breck32@breckyunits.com:/var/www/html/ --exclude '.git' --exclude 'deploy' --exclude 'README' --exclude 'posts.php'

This one line will do a fast, incremental file transfer from my local directory to the web server.

I saved this file as "deploy", and then run it by typing:

./deploy

The exclude option will exclude a file or directory from being synced.

Here's an explanation about the options I use. I use rsync with these options:

rsync -arvuz /src/foo/ /dest/foo

This will copy the contents of /src/foo/ into /dest/foo. Options explained:

  • a: archive mode (keep owner and permissions)
  • r: recurse into directories
  • v: increase verbosity
  • u: update only (don't overwrite newer files - if you have edited files in destination)
  • z: compress file data

Notes

  1. If you are using a different port for SSH, say 333 do this: --rsh="ssh -p333"



Edit

Get VIM Working in Cygwin

November 20, 2009

I had trouble getting VIM behaving as expected under Cygwin. Specifically, the backspace and arrow keys were behaving badly in insert mode.

The solution was dead simple. In your root directory just run:

touch .vimrc

Links

  • Found this solution in the comment from Gregg vimrc



Edit

On your server:

vim /etc/ssh/sshd_config

Make sure allow rsa and the other RSA lines are uncommented

service sshd restart

Make sure the file ~/.ssh/authorized_keys exists(if it doesn't, make that directory and touch that file).

Run this on your client computer:

ssh-keygen -t rsa

(hit enter a bunch of times)

cat ~/.ssh/id_rsa.pub | ssh YOURUSERNAME@YOURDOMAIN.com "cat - >> ~/.ssh/authorized_keys"

Bam! Login without a password!

This is my setup on Fedora VM's running at slicehost.

If you are having problems

  1. Run sshd with the verbose option. -v
  1. The problem is likely due to bad file permissions. The owner of the directory .ssh should be your user/group. The file permissions on it should be: 755. (I think) authorized_keys should be the same owner/group and file permissions should be: 644



Edit

About this Blog

November 18, 2009

I constantly want to share little coding tidbits, snippets, or stories on how to solve specific little coding problems. Instead of clogging up the main blog with these shorter, more specific technical posts, I decided to launch a second blog.

This is it.




Edit



Built with Scroll v148.2.0