UNIX and DOS: Binary vs. Ascii, or CR-LF issues

The ASCII code is a standardized computer code which corresponds every letter, number, punctuation mark, etc. to a number between 0 and 255. This is the way computers store text files: each character in the file corresponds to a byte (8 bits, 2^8 = 256 possible combinations).

The line feed (LF) has ASCII code 10, and the carriage return (CR) has ASCII code 13. When computers first proliferated, there was some debate over how the end of lines should be stored in the file. To this day, UNIX systems use only the LF, while DOS systems use CRLF. That is, if your file looks like this: 

he
llo
UNIX would store this file as: 104 - 101 - 10 - 108 - 108 - 111
while DOS would store this file as: 104 - 101 - 13 - 10 - 108 - 108 - 111.

I've highlighted red the ASCII bytes which designate the new line.

It is now widely acknowledged that the DOS system is more cumbersome; after all why do you need to include two bytes if you can get the same information with just one byte?

VMS follows the UNIX convention, whereas Windows (obviously) follows the DOS convention.

Now, this CRLF/LF discrepancy can cause problems when FTP-ing files between computers. You probably know that there are two modes of transferring files in FTP: binary and ASCII. In binary mode, files transfer exactly from one machine to the other with no file conversion. In ASCII mode, the FTP program tries to fix the file so that new lines are converted as appropriate. Both methods have their share of problems, depending on what you want to do. Let's take an example....

You have a text file on a UNIX system; you want to transfer it to the DOS system.

When FTP-ing a file from UNIX to DOS under ascii mode: it changes all instances of LF to CRLF (i.e. inserts the extra byte of value 13 before the byte of value 10). Now you can read the file on your Windows/DOS system and everything will read fine.

But what if you had transferred under binary mode? It would leave the LF as is, leaving the file unchanged. The Windows program Notepad can't handle it correctly, and you get black boxes instead of new lines. Open it in MS Word or Wordpad, however, and these two programs are smart enough to realize that the LF actually means a newline.

Now, let's go the other way....

When FTP-ing a file from DOS to UNIX under ascii mode: it changes all instances of CRLF to LF. All is well again.

Let's say you transferred the file from VMS to DOS under binary mode, and want to transfer it right back, also under binary mode. Seems like since binary mode doesn't alter the file, the new file on VMS system should match the file that you started with, right? Wrong! That's because VMS changes the record format and attributes when you transfer as binary mode, because VMS thinks you're transferring a program or image. Even though the bytes don't change, when you try to read the file, VMS doesn't realize it's a text file and displays it wrong.

Some other things to note.