Free Downloads, Community Forum,
FAQs and Developer Resources


Make /Tools Your Home | Link to us

Today's posts | Posts since last visit | Most Active Topics

All Forums Register Login Search Subscriptions My Profile Inbox
Tool Warehouse FAQs Resources Help Member List Address Book Logout

DOS vs. Unix line endings

 
Logged in as: Guest
Users viewing this topic: none
  Printable Version
All Forums >> [SFU / Interix / SUA Technology] >> SFU / Interix - Getting Started >> DOS vs. Unix line endings Page: [1] 2   next >   >>
Login
Message << Older Topic   Newer Topic >>
DOS vs. Unix line endings - May 10, '06, 12:47:32 AM   
dpchrist

 

Posts: 9
Joined: May 9, '06,
Status: offline
SFU/Interix - Getting Started:

I'm an SFU newbie coming from Cygwin. I have installed SFU 3.5, SFU bash, run chsh, copied/edited the SFU shell shortcut, and set my $HOME environment variable to match my Cygwin home. When I launch SFU bash, it attempts to run .bash_profile:

Welcome to the Interix UNIX utilities.

DISPLAY=localhost:0.0
: command not found
bash: /dev/fs/C/cygwin/home/dpchrist/.bash_profile: line 95: syntax error: unexpected end of file
bash-3.00$


Running some simple tests, it appears that SFU uses Unix line endings. I prefer DOS line endings on Microsoft platforms, and have configured Cygwin to use DOS line endings.

Is there a way to configure SFU to use DOS line endings?

TIA,

David
Post #: 1
RE: DOS vs. Unix line endings - May 10, '06, 3:37:08 AM   
Rodney

 

Posts: 3728
Joined: Jul. 9, '02,
From: /Tools lab
Status: offline
Nope.
Some specific applications/API that read text files will read both ways, but most not.
Use the flip utility to set the files to Unix endings (or Mac or DOS) [flip -u].

(in reply to dpchrist)
Post #: 2
RE: DOS vs. Unix line endings - May 10, '06, 10:04:56 PM   
dpchrist

 

Posts: 9
Joined: May 9, '06,
Status: offline
Line endings need to be consistant across all text files on a system. Line endings are a fundamental design decision of a given platform, and intertwined throughout the implementation. Going against convention only invites grief.

3rd party tools which impose a requirement for non-native line endings will be of limited use.

Has anyone figured out how to get SFU Bash to work with text files having native line endings?

Has Interix imposed a decision of "Unix line endings only" upon its developers and porters?

David

(in reply to Rodney)
Post #: 3
RE: DOS vs. Unix line endings - May 11, '06, 9:01:58 PM   
markfunk

 

Posts: 673
Joined: Mar. 31, '03,
Status: offline
Line endings need to be consistant across all text files on a system.
Correct. Interix is a UNIX environement subsystem designed to a) be a real UNIX system
and b) run concurrently with the Win32 subsystem.
As such, all the Interix utils and libraries comes from original UNIX source code
and thus by definition only deals with UNIX text file formats.

Line endings are a fundamental design decision of a given platform, and intertwined throughout the implementation. Going against convention only invites grief.
Correct, the design decision for Interix was to use the UNIX convention.
Any other convention would have invited potential unexpected behaviour (e.g "grief") to the
UNIX application programmer.
The decision to support both DOS and UNIX text formats has to be an application decision.

The benefit of the Interix subsystem is to allow end-users to bring their UNIX apps to Windows.
If they want their UNIX app to interoperate with the native win32 conventions,
then they need to design/build that into their app.
That is a end-user design decision, not an Interix environment decision.

That said, there are many std Interix applications (ksh, awk, "vi -W", ... ) that have been
modified to deal with DOS text format.

I use Cygwin and Interix interchangably on my systems and I haven't had
to work around any problems with DOS vs. UNIX text formats.
I use bash on cygwin and ksh on Interix, both sharing the same .profile and .environ.

The Interix bash that I have (from Interopsystems), does support both text formats.
I don't have need to use Interix bash very often.

3rd party tools which impose a requirement for non-native line endings will be of limited use
Interix is not a 3rd party tool.

Has anyone figured out how to get SFU Bash to work with text files having native line endings?
The bash that I have (2.05b from Interopsystems), does seem to support both text formats.

Has Interix imposed a decision of "Unix line endings only" upon its developers and porters?
Not directly. The decision in Interix was to use and support std/historical UNIX library source code.
So, the expectation when using UNIX api's is to get the same behaviour you'd expect on any other UNIX system.
Interix imposes no limitation on the application that may want to support additional
features (like DOS text formats).

(in reply to dpchrist)
Post #: 4
RE: DOS vs. Unix line endings - May 12, '06, 2:00:25 AM   
dpchrist

 

Posts: 9
Joined: May 9, '06,
Status: offline
Rodney and markfunk -- thanks for your help. :-)



Interix is a UNIX environement subsystem designed to a) be a real UNIX system
and b) run concurrently with the Win32 subsystem.
As such, all the Interix utils and libraries comes from original UNIX source code
and thus by definition only deals with UNIX text file formats.


I am looking for a stable, versioned, complete, documented, and supported GNU toolchain for Windows that offers maximum interoperability with the native Windows environment. My primary uses are interactive Bash shell usage and Perl scripting. I prefer ActiveState Perl. I develop and maintain a small number of applications programs and system administration utilities. Free is preferable, but I have used commercial products in the past (MKS Toolkit, MicroCross).

Is Interix the wrong tool for me?


the design decision for Interix was to use the UNIX convention.
The decision in Interix was to use and support std/historical UNIX library source code.
So, the expectation when using UNIX api's is to get the same behaviour you'd expect on any other UNIX system.
Interix imposes no limitation on the application that may want to support additional
features (like DOS text formats).


Compromises are unavoidable. I would have prefered a decision set that solves the line ending issue easily. (Having it slap me in the face as soon as I start Bash is not motivating...)



The decision to support both DOS and UNIX text formats has to be an application decision.
The benefit of the Interix subsystem is to allow end-users to bring their UNIX apps to Windows.
If they want their UNIX app to interoperate with the native win32 conventions,
then they need to design/build that into their app.
That is a end-user design decision, not an Interix environment decision.


Ouch. I used to know how to set text/binary mode switches in open() calls way back when I programmed in C, but now I'm just lazy and let Perl figure it out for me. :-)

Line endings are one thing, drive identifiers and path separators follow shortly, and the list is only going to get uglier the deeper one goes. Again, Cygwin and Perl have useful mechanisms to ease the pain, but I agree that there's going to be issues and areas where it is not practical to try to solve or work around them in the programming environment.

But, Cygwin has already solved the line ending issue. Interix might want to be more competitive.



That said, there are many std Interix applications (ksh, awk, "vi -W", ... ) that have been
modified to deal with DOS text format.


I trust patches have been submitted to the authors/ maintainers?



I use Cygwin and Interix interchangably on my systems and I haven't had
to work around any problems with DOS vs. UNIX text formats.


What line endings are you using in Interix and what line endings are you using in Cygwin?



I use bash on cygwin and ksh on Interix, both sharing the same .profile and .environ.


I use Bash and CVS on Cygwin, Debian GNU/Linux, and Slackware, all sharing various text files. CVS sets the line endings to system native for me.



The Interix bash that I have (from Interopsystems), does support both text formats.
I don't have need to use Interix bash very often.
...
The bash that I have (2.05b from Interopsystems), does seem to support both text formats.


Where do I get that version of Bash? Or, can I configure my version to use DOS line endings?

bash-3.00$ pkg_info | grep bash
bash-3.0.0.6-bin   Version 3.0.0.6 of bash for Interix.




Interix is not a 3rd party tool.


So, Interix is a product developed by Interop Systems, which is a subsidiary/ division/ whatever of Microsoft?



David

(in reply to markfunk)
Post #: 5
RE: DOS vs. Unix line endings - May 12, '06, 4:16:09 AM   
Rodney

 

Posts: 3728
Joined: Jul. 9, '02,
From: /Tools lab
Status: offline
I will respond to some of your questions:
quote:

So, Interix is a product developed by Interop Systems, which is a subsidiary/ division/ whatever of Microsoft?


No. Interix was originally developed under the name OpenNT by Softway Systems. Microsoft bought Interix in 1999.
Interop Systems is not owned by Microsoft. Mark and myself were part of the team that developed it at Softway.

quote:

I use Bash and CVS on Cygwin, Debian GNU/Linux, and Slackware, all sharing various text files. CVS sets the line endings to system native for me.


CVS is on Interix sets line endings to native. Interix and Win32 are peers over the NT kernel.

quote:

Ouch. I used to know how to set text/binary mode switches in open() calls way back when I programmed in C, but now I'm just lazy and let Perl figure it out for me. :-)


It has nothing to do with open(). It has to do with how a line is parsed by the functions that are
reading/writing the file. Unix lines end in <newline>. DOS/Win32 lines end in <return><newline>.

quote:

I am looking for a stable, versioned, complete, documented, and supported GNU toolchain for Windows that offers maximum interoperability with the native Windows environment. My primary uses are interactive Bash shell usage and Perl scripting. I prefer ActiveState Perl. I develop and maintain a small number of applications programs and system administration utilities. Free is preferable, but I have used commercial products in the past (MKS Toolkit, MicroCross).


The commands and utilities with Interix meet POSIX and SUS standards. They are BSD based.
There are a number of GNU utilities available as binaries. You can always build them yourself too.
The original Perl is available for you to use too. But if you want to drive the ActiveState perl
from scritps that can done. The download of Interix and its use is free. The additional binary
package from here are also free.

I'll note that there are several additional editors available, such as nedit, that work
with both types of line endings without harm (leaving it as is). Actually all of the Interix
editors won't change line ending unless you explicitly set or do it. The same cannot be said for
Win32 editors.

(in reply to dpchrist)
Post #: 6
RE: DOS vs. Unix line endings - May 12, '06, 8:03:35 PM   
dpchrist

 

Posts: 9
Joined: May 9, '06,
Status: offline
Rodney wrote:
> Interix and Win32 are peers over the NT kernel.
> ...
> The commands and utilities with Interix meet POSIX and SUS standards.
> They are BSD based.

Thanks for the reply. :-)

Okay. That clarifies much.


markfunk wrote:
> The Interix bash that I have (from Interopsystems), does support both
> text formats. I don't have need to use Interix bash very often.
> ...
> The bash that I have (2.05b from Interopsystems), does seem to
> support both text formats.

I wrote:
> Where do I get that version of Bash? Or, can I configure my version
> to use DOS line endings?

I am unable to use SFU until I resolve the line ending issue for Bash. Does anybody know the answer?


TIA,

David

(in reply to Rodney)
Post #: 7
RE: DOS vs. Unix line endings - May 13, '06, 4:47:44 AM   
Rodney

 

Posts: 3728
Joined: Jul. 9, '02,
From: /Tools lab
Status: offline
The code Mark makes reference to has been blocked out as cygwin only in the source.
Earlier version didn't have it blocked out; it was on all platforms.
Why the change? I don't know.

The source can be modified to for other systems to do it.
I won't be able to get to it until Monday at the earliest.
But if your script is portable then try running it with ksh.

(in reply to dpchrist)
Post #: 8
RE: DOS vs. Unix line endings - May 13, '06, 4:35:54 PM   
dpchrist

 

Posts: 9
Joined: May 9, '06,
Status: offline
Rodney wrote:
> The code Mark makes reference to has been blocked out as cygwin only
> in the source. Earlier version didn't have it blocked out; it was on
> all platforms. Why the change? I don't know.
> The source can be modified to for other systems to do it.
> I won't be able to get to it until Monday at the earliest.

Thanks for the reply. :-)

Okay. Maybe Mark remembers and will post to this thread.


> But if your script is portable then try running it with ksh.

I used ksh on MKS TK years ago and really liked it, but when launch the SFU ksh I immediately trip over the cursor, Home, End, and Delete keys, and next over its (apparent) lack of history. Maybe these things can be adjusted, but I now use Bash everywhere and decided to try that approach on SFU. When I start SFU Bash, it isn't making it past my .bash_profile:

Welcome to the Interix UNIX utilities.

DISPLAY=localhost:0.0
: command not found
bash: /dev/fs/C/cygwin/home/dpchrist/.bash_profile: line 95: syntax error: unexp
ected end of file
bash-3.00$


Using Cygwin to change the line endings:

2006-05-13 13:31:22 dpchrist@p3533 ~
$ dos2unix .bash_profile
.bash_profile: done.
2006-05-13 13:31:27 dpchrist@p3533 ~
$ dos2unix .bashrc
.bashrc: done.


And trying SFU Bash again:

Welcome to the Interix UNIX utilities.

DISPLAY=localhost:0.0
2006-05-13 13:32:09 dpchrist@p3533 ~
$


Yup, the problem is SFU Bash can't handle DOS line endings. Browsing and searching "man bash" doesn't reveal and clues for changing this.


So, I'm dead in the water with SFU.


David

(in reply to Rodney)
Post #: 9
RE: DOS vs. Unix line endings - May 14, '06, 3:08:44 AM   
Rodney

 

Posts: 3728
Joined: Jul. 9, '02,
From: /Tools lab
Status: offline
> Okay. Maybe Mark remembers and will post to this thread.

unlikely -- the code change was done by bash maintainers.
That's why I posted that a code change needs to be done.

The command for changing file endings with Interix is flip.
It works with Unix, DOS and Mac line endings.

Ksh has history. The default editing mode is 'vi' (as per the standards).
You can switch the editing mode to 'emacs' with the VISUAL env var (see the man page).

(in reply to dpchrist)
Post #: 10
RE: DOS vs. Unix line endings - May 14, '06, 12:18:45 PM   
markfunk

 

Posts: 673
Joined: Mar. 31, '03,
Status: offline
I am looking for a stable, versioned, complete, documented, and supported GNU toolchain for Windows that offers maximum interoperability with the native Windows environment.

Is Interix the wrong tool for me?

Probably. If you want GNU, then Cygwin is probably the most popular.
If you want max. interop with native Windows, then you want Win32 applications.
Like Cygwin and MKS toolkit and Uwin, which are Win32 apps that provide/emulate UNIX functionality.

Line endings are one thing, drive identifiers and path separators follow shortly, and the list is only going to get uglier the deeper one goes. Again, Cygwin and Perl have useful mechanisms to ease the pain,
Correct. You definitely won't be happy with Interix.
Cygwin and ActiveState perl have the luxury to run loose and free. They just try to provide
UNIX functionality as much as possible - where it makes sense in a Win32 environment.
Interix is first and foremost a real UNIX environment. It has a requirement to be UNIX conforming in all aspects.

Interix isn't for everyone. It doesn't clam to be. It was never designed/intended to compete with
the likes of Cygwin and MKS toolkit, unless you wanted "real UNIX".

What line endings are you using in Interix and what line endings are you using in Cygwin?
I don't really think about it. It all just works for me.
Looking at one of my sytems that started with cygwin, all the line endings seem to be DOS format.


The really hard part of maintaining my shell scripts between these environments is with
the pathname syntax. I tend to use win32 pathnames but call a special cygpath wrapper script
which detects if I'm in Cygwin, MKS or Interix and then provides the correctly formated pathname.

(in reply to Rodney)
Post #: 11
RE: DOS vs. Unix line endings - May 16, '06, 1:41:03 AM   
Rodney

 

Posts: 3728
Joined: Jul. 9, '02,
From: /Tools lab
Status: offline
An updated bash is now available (version 3.0.0.7) that works with "\n" and "\r\n" line endings.

To get the lastest do:
    pkg_update -L bash


(Note: sh/ksh and csh/tcsh already handled this because they do this
on all platforms already <-- just for the curious and/or nervous.)

Just to expand on the pathname syntax: there are two utilities (based on two API's by the same
names) that convert pathnames between Win32 and Interix -- winpath2unix & unixpath2win.

I'll also note that you can with Interix 5.2 and later (W2K3/R2 and later) you can create a mixed mode
binary that can have Interix and Win32 parts in one. (i.e. Oracle Win32 libs linked to Interix code).

(in reply to markfunk)
Post #: 12
RE: DOS vs. Unix line endings - May 17, '06, 12:30:46 AM   
dpchrist

 

Posts: 9
Joined: May 9, '06,
Status: offline
markfunk wrote:
> If you want GNU, then Cygwin is probably the most popular. If you
> want max. interop with native Windows, then you want Win32
> applications. Like Cygwin and MKS toolkit and Uwin, which are Win32
> apps that provide/emulate UNIX functionality.
...
> Interix is first and foremost a real UNIX environment. It has a
> requirement to be UNIX conforming in all aspects.
> Interix isn't for everyone. It doesn't clam to be. It was never
> designed/intended to compete with the likes of Cygwin and MKS
> toolkit, unless you wanted "real UNIX".

Thanks for the reply. :-)

I'm familiar with Cygwin and MKS Toolkit. I'll try UWIN and see how well it meets my needs.


Rodney wrote:
> An updated bash is now available (version 3.0.0.7) that works
> with "\n" and "\r\n" line endings.

Thanks for the reply. :-)

I upgraded Bash, but it still fails:

SFU Bash:
Welcome to the Interix UNIX utilities.

DISPLAY=localhost:0.0
: command not found
bash: /dev/fs/C/cygwin/home/dpchrist/.bash_profile: line 95: syntax error: unexpected end of file
bash-3.00$


Cygwin Bash:
2006-05-16 21:14:36 dpchrist@p3533 ~
$ dos2unix .bash_profile
.bash_profile: done.
2006-05-16 21:14:37 dpchrist@p3533 ~
$ dos2unix .bashrc
.bashrc: done.


SFU Bash:
Welcome to the Interix UNIX utilities.

DISPLAY=localhost:0.0
2006-05-16 21:14:48 dpchrist@p3533 ~
$

$ pkg_info | grep bash
bash-3.0.0.7-bin   Version 3.0.0.7 of bash for Interix.



It appears that Interix Bash 3.0.0.7 is not working with DOS line endings on my computer.


I appreciate the quick turn-around and attempted feature addition, but please don't break your code base (or your users' applications) on my account.


David

(in reply to Rodney)
Post #: 13
RE: DOS vs. Unix line endings - May 17, '06, 1:10:13 AM   
Rodney

 

Posts: 3728
Joined: Jul. 9, '02,
From: /Tools lab
Status: offline
What's on line 95 ?
It made it through the previous 94 lines without a message.

(in reply to dpchrist)
Post #: 14
RE: DOS vs. Unix line endings - May 17, '06, 10:59:54 AM   
woehlkmp

 

Posts: 102
Status: offline
quote:

Just to expand on the pathname syntax: there are two utilities (based on two API's by the same
names) that convert pathnames between Win32 and Interix -- winpath2unix & unixpath2win.

It may be worth noting that I ran into problems with those being less "robust" than they are on Cygwin. Specifically, cygpath will "figure out" what format the input path is in and give it to you in the requested format. The winpath2unix / unixpath2win tools make assumptions that make them less generically useful.

I wrote a C program for Interix that emulates cygpath's "intelligence". I might be able to get you an updated version if that looks useful (I know I played around with it a bunch since that post).

(in reply to Rodney)
Post #: 15
RE: DOS vs. Unix line endings - May 17, '06, 11:13:48 PM   
dpchrist

 

Posts: 9
Joined: May 9, '06,
Status: offline
Rodney wrote:
> What's on line 95 ?
> It made it through the previous 94 lines without a message.

Nothing:

2006-05-17 20:12:53 dpchrist@p3533 ~
$ wc .bash_profile
94 244 2217 .bash_profile


David

(in reply to Rodney)
Post #: 16
RE: DOS vs. Unix line endings - May 18, '06, 12:11:09 AM   
Rodney

 

Posts: 3728
Joined: Jul. 9, '02,
From: /Tools lab
Status: offline
Okay, so there are 94 lines and it complains at line 95.
In the file what are all of the characters of the last line
in octal (using the o/p from the od utility) ?
I ask since I think there might be something there/not-there
that is non-visible.

(in reply to dpchrist)
Post #: 17
RE: DOS vs. Unix line endings - May 19, '06, 12:13:11 AM   
dpchrist

 

Posts: 9
Joined: May 9, '06,
Status: offline
Rodney wrote:
> Okay, so there are 94 lines and it complains at line 95.
> In the file what are all of the characters of the last line
> in octal (using the o/p from the od utility) ?
> I ask since I think there might be something there/not-there
> that is non-visible.

$ nl -ba .bash_profile | tail
    85  if [ -d $HOME/cvs/man ] ; then
    86      MANPATH=$MANPATH:$HOME/cvs/man
    87  fi
    88  if [ -d $HOME/man ] ; then
    89      MANPATH=$MANPATH:$HOME/man
    90  fi
    91  if [ -d $HOME/share/man ] ; then
    92      MANPATH=$MANPATH:$HOME/share/man
    93  fi
    94  export MANPATH=:$HOME/local/man

<CVSENV><CVSENV>2006-05-18 21:12:36 dpchrist@p3533 ~
$ dump .bash_profile | tail
000820   A   T   H   =   $   M   A   N   P   A   T   H   :   $   H   O
000830   M   E   /   m   a   n  \r  \n   f   i  \r  \n   i   f       [
000840       -   d       $   H   O   M   E   /   s   h   a   r   e   /
000850   m   a   n       ]       ;       t   h   e   n  \r  \n
000860           M   A   N   P   A   T   H   =   $   M   A   N   P   A
000870   T   H   :   $   H   O   M   E   /   s   h   a   r   e   /   m
000880   a   n  \r  \n   f   i  \r  \n   e   x   p   o   r   t       M
000890   A   N   P   A   T   H   =   :   $   H   O   M   E   /   l   o
0008a0   c   a   l   /   m   a   n  \r  \n
0008a9



David

(in reply to Rodney)
Post #: 18
RE: DOS vs. Unix line endings - May 19, '06, 12:55:38 AM   
Rodney

 

Posts: 3728
Joined: Jul. 9, '02,
From: /Tools lab
Status: offline
Thanks. I don't see the "smoking gun" that I was hoping for (invisible character).

I've tried running the current bash against scripts with/without \r on all,
some and none of the line -- it's working fine. I tried it without a few \n even.
mmm, it does say "syntax error, unexpected end of file". Perhaps there is one line
that \r leaves something open...
ah, by copying what you have I can now replicate the error.
Don't know what it is just yet.

(in reply to dpchrist)
Post #: 19
RE: DOS vs. Unix line endings - May 19, '06, 6:05:15 PM   
woehlkmp

 

Posts: 102
Status: offline
Ok, are we overlooking the obvious?
"Unexpected end of file" usually means "Ok, you told me 'if ...'; I'm still waiting to see 'fi'", and things of that nature. Have you verified that your script does not have unterminated constructs? (A text editor with code folding can help with that... as I can attest to from having just written such a script two minutes ago .)

(in reply to Rodney)
Post #: 20
Page:   [1] 2   next >   >>
All Forums >> [SFU / Interix / SUA Technology] >> SFU / Interix - Getting Started >> DOS vs. Unix line endings Page: [1] 2   next >   >>
Jump to:





New Messages No New Messages
Hot Topic w/ New Messages Hot Topic w/o New Messages
Locked w/ New Messages Locked w/o New Messages
 Post New Thread
 Reply to Message
 Post New Poll
 Submit Vote
 Delete My Own Post
 Delete My Own Thread
 Rate Posts


Search All Forums -

Advanced search


SPONSORS



Forum Software © ASPPlayground.NET Advanced Edition 2.5 ANSI

0.141