In my shell script I want to read csv file line by line, and remove prefix and suffix from one of the column.

here is a sample input.csv:, user1, "2zG7CBKp5d4z", user2, "9vy6J,M2A3b6", user3, "V5q,8mvz#$ac"

here is my code:



while IFS="," read f1 f2 f3
        echo "$f1" "$f2" "$f3"

done <"$filename"

In the script with while I want to ssh to host ($f1) with user ($f2) and password ($f3), but have to remove the double-quote from f3. The ssh part will be implemented later. The prefix removal works as expected, but the suffix does NOT. Anybody can tell me why ?

  • 1
    When I tried this (with a here-doc instead of separate file), it's the prefix removal that fails, since the third field begins with a space before ". – Toby Speight Jun 7 at 9:51

Removing the trailing " works for me. I suspect that your lines, and therefore f3, do not end with " but with \r – the first part of a windows line break (\r\n), opposed to unix line breaks (\n).

Convert your file using dos2unix original or tr -d \\r < original > converted.

By the way, the last field is only read correctly because it is the last. Usually read would interpret the comma inside the last field as the beginning of a new field – read does not give special meaning to the quotes. However, read will read all remaining fields into the last variable given, in this case f3.

  • Ohhh yes... the line break was the problem. Changing it from \r\n to unix line breaks \n solve the problem. Not required to ignore \r – jtarjanyi Jun 7 at 10:02

If the last field length is constant, cut efficiently gets rid of the quotes:

rev input.csv | cut -c2-13,15- | rev

If all the field lengths are constant, there's no need for rev:

cut -c-20,22-33 input.csv
  • Unlike the OP code, this answer doesn't change "," to " " , but if the first and last fields are fixed lengths, then cut could manage that as well using the same method. – agc Jun 7 at 14:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.