[perl #125619] Documentation of byte I/O

[Rob Dixon (via RT)]

# New Ticket Created by Rob Dixon
# Please include the string: [perl #125619]
# in the subject line of all future correspondence about this issue.
# <URL: https://rt.perl.org/Ticket/Display.html?id=125619 >


This is a bug report for perl from ***@gmail.com,
generated with the help of perlbug 1.40 running under perl 5.22.0.


-----------------------------------------------------------------

I recently read "perldoc bytes" and saw

This pragma reflects early attempts to incorporate Unicode into
perl (sic.) and has since been superseded

I think it is a mistake to fail to say /what/ has superseded it, and
the perluni* pods aren't something I should expect to look at if I'm
not using Unicode

In any statement that deprecates "bytes", I think something should
be said about "utf8", which seems to be its dual but is not

In short, I believe the perluni* should have been
"perlcharacterencoding", as there is no "perlascii" or "perllatin1"

The journey into the promised land of Unicode has been arduous
enough as it is, and the perluni* pods are a major achievement, so I
don't really expect all of that to be ripped apart on my whim. But
perhaps we could do with something like my "perlcharacterencoding"
as an entry point to all of the above?

As you can tell, these are infant musings without any coherent plan.
But I have recently been asked how to enable byte semantics on a
Perl input stream, and I was troubled to find that I was neither
certain nor able to locate the documentation that told me. I suspect
the answer is that file handles have byte semantics by default, and
can be opened that way explicitly with a mode of :raw or using
binmode, but I am far from sure

Or perhaps I am just asking for a better index into the perldoc tomes

Thank you for reading

-----------------------------------------------------------------
---
Flags:
category=docs
severity=wishlist
---
Site configuration information for perl 5.22.0:

Configured by strawberry-perl at Mon Jun 1 20:06:45 2015.

Summary of my perl5 (revision 5 version 22 subversion 0) configuration:

Platform:
osname=MSWin32, osvers=6.3, archname=MSWin32-x86-multi-thread-64int
uname='Win32 strawberry-perl 5.22.0.1 #1 Mon Jun 1 20:04:50 2015 i386'
config_args='undef'
hint=recommended, useposix=true, d_sigaction=undef
useithreads=define, usemultiplicity=define
use64bitint=define, use64bitall=undef, uselongdouble=undef
usemymalloc=n, bincompat5005=undef
Compiler:
cc='gcc', ccflags =' -s -O2 -DWIN32 -DPERL_TEXTMODE_SCRIPTS
-DPERL_IMPLICIT_CONTEXT -DPERL_IMPLICIT_SYS -fwrapv
-fno-strict-aliasing -mms-bitfields',
optimize='-s -O2',
cppflags='-DWIN32'
ccversion='', gccversion='4.9.2', gccosandvers=''
intsize=4, longsize=4, ptrsize=4, doublesize=8,
byteorder=12345678, doublekind=3
d_longlong=define, longlongsize=8, d_longdbl=define,
longdblsize=8, longdblkind=3
ivtype='long long', ivsize=8, nvtype='double', nvsize=8,
Off_t='long long', lseeksize=8
alignbytes=8, prototype=define
Linker and Libraries:
ld='g++', ldflags ='-s -L"C:\STRAWB~1\perl\lib\CORE" -L"C:\STRAWB~1\c\lib"'
libpth=C:\STRAWB~1\c\lib C:\STRAWB~1\c\i686-w64-mingw32\lib
C:\STRAWB~1\c\lib\gcc\i686-w64-mingw32\4.9.2
libs=-lmoldname -lkernel32 -luser32 -lgdi32 -lwinspool -lcomdlg32
-ladvapi32 -lshell32 -lole32 -loleaut32 -lnetapi32 -luuid -lws2_32
-lmpr -lwinmm -lversion -lodbc32 -lodbccp32 -lcomctl32
perllibs=-lmoldname -lkernel32 -luser32 -lgdi32 -lwinspool
-lcomdlg32 -ladvapi32 -lshell32 -lole32 -loleaut32 -lnetapi32 -luuid
-lws2_32 -lmpr -lwinmm -lversion -lodbc32 -lodbccp32 -lcomctl32
libc=, so=dll, useshrplib=true, libperl=libperl522.a
gnulibc_version=''
Dynamic Linking:
dlsrc=dl_win32.xs, dlext=xs.dll, d_dlsymun=undef, ccdlflags=' '
cccdlflags=' ', lddlflags='-mdll -s -L"C:\STRAWB~1\perl\lib\CORE"
-L"C:\STRAWB~1\c\lib"'


---
@INC for perl 5.22.0:
C:/Strawberry/perl/site/lib/MSWin32-x86-multi-thread-64int
C:/Strawberry/perl/site/lib
C:/Strawberry/perl/vendor/lib
C:/Strawberry/perl/lib
.

---
Environment for perl 5.22.0:
HOME (unset)
LANG (unset)
LANGUAGE (unset)
LD_LIBRARY_PATH (unset)
LOGDIR (unset)
PATH=C:\Python27\;C:\Python27\Scripts;C:\Program Files
(x86)\Common Files\Intel\Shared
Files\cpp\bin\Intel64;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Strawberry\c\bin;C:\Strawberry\perl\site\bin;C:\Strawberry\perl\bin;C:\ffmpeg\bin;C:\Program
Files (x86)\Git\cmd;C:\Program Files (x86)\NVIDIA
Corporation\PhysX\Common;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;E:\Perl\source;C:\Program
Files (x86)\EaseUS\Todo Backup\bin\x64\
PERL_BADLANG (unset)
SHELL (unset)

[Chas. Owens]

Would it be correct to say that explicit encoding using the Encode module
has replaced the bytes pragma's implicit treatment of strings as bytes of
unknown encoding?

expected result of the bytes pragma:
2 [0xc3 0xa9]
unexpected results when string is not in the expected encoding
1 [0xe9]
explicit encoding yields the expected results
2 [0xc3 0xa9]
even when the string isn't in the expected encoding
2 [0xc3 0xa9]


#!/usr/bin/perl

use strict;
use warnings;
use utf8;

use Encode qw/encode/;

my $utf8 = "é";

print "expected result of the bytes pragma:\n";
{
use bytes;


my $length = length $utf8;
my @bytes = map { sprintf "0x%02x", ord } split //, $utf8;

print "$length [@bytes]\n";
}

print "unexpected results when string is not in the expected encoding\n";
my $latin1 = encode("Latin1", $utf8);
{
use bytes;

my $length = length $latin1;
my @bytes = map { sprintf "0x%02x", ord } split //, $latin1;

print "$length [@bytes]\n";
}

print "explicit encoding yields the expected results\n";
{
my $raw = encode('UTF-8', $utf8);
my $length = length $raw;
my @bytes = map { sprintf "0x%02x", ord } split //, $raw;

print "$length [@bytes]\n";
}

print "even when the string isn't in the expected encoding\n";
{
my $raw = encode('UTF-8', $latin1);
my $length = length $raw;
my @bytes = map { sprintf "0x%02x", ord } split //, $raw;

print "$length [@bytes]\n";
}

Post by Rob Dixon (via RT)
# New Ticket Created by Rob Dixon
# Please include the string: [perl #125619]
# in the subject line of all future correspondence about this issue.
# <URL: https://rt.perl.org/Ticket/Display.html?id=125619 >
generated with the help of perlbug 1.40 running under perl 5.22.0.
-----------------------------------------------------------------
I recently read "perldoc bytes" and saw
This pragma reflects early attempts to incorporate Unicode into
perl (sic.) and has since been superseded
I think it is a mistake to fail to say /what/ has superseded it, and
the perluni* pods aren't something I should expect to look at if I'm
not using Unicode
In any statement that deprecates "bytes", I think something should
be said about "utf8", which seems to be its dual but is not
In short, I believe the perluni* should have been
"perlcharacterencoding", as there is no "perlascii" or "perllatin1"
The journey into the promised land of Unicode has been arduous
enough as it is, and the perluni* pods are a major achievement, so I
don't really expect all of that to be ripped apart on my whim. But
perhaps we could do with something like my "perlcharacterencoding"
as an entry point to all of the above?
As you can tell, these are infant musings without any coherent plan.
But I have recently been asked how to enable byte semantics on a
Perl input stream, and I was troubled to find that I was neither
certain nor able to locate the documentation that told me. I suspect
the answer is that file handles have byte semantics by default, and
can be opened that way explicitly with a mode of :raw or using
binmode, but I am far from sure
Or perhaps I am just asking for a better index into the perldoc tomes
Thank you for reading
-----------------------------------------------------------------
---
category=docs
severity=wishlist
---
Configured by strawberry-perl at Mon Jun 1 20:06:45 2015.
osname=MSWin32, osvers=6.3, archname=MSWin32-x86-multi-thread-64int
uname='Win32 strawberry-perl 5.22.0.1 #1 Mon Jun 1 20:04:50 2015 i386'
config_args='undef'
hint=recommended, useposix=true, d_sigaction=undef
useithreads=define, usemultiplicity=define
use64bitint=define, use64bitall=undef, uselongdouble=undef
usemymalloc=n, bincompat5005=undef
cc='gcc', ccflags =' -s -O2 -DWIN32 -DPERL_TEXTMODE_SCRIPTS
-DPERL_IMPLICIT_CONTEXT -DPERL_IMPLICIT_SYS -fwrapv
-fno-strict-aliasing -mms-bitfields',
optimize='-s -O2',
cppflags='-DWIN32'
ccversion='', gccversion='4.9.2', gccosandvers=''
intsize=4, longsize=4, ptrsize=4, doublesize=8,
byteorder=12345678, doublekind=3
d_longlong=define, longlongsize=8, d_longdbl=define,
longdblsize=8, longdblkind=3
ivtype='long long', ivsize=8, nvtype='double', nvsize=8,
Off_t='long long', lseeksize=8
alignbytes=8, prototype=define
ld='g++', ldflags ='-s -L"C:\STRAWB~1\perl\lib\CORE"
-L"C:\STRAWB~1\c\lib"'
libpth=C:\STRAWB~1\c\lib C:\STRAWB~1\c\i686-w64-mingw32\lib
C:\STRAWB~1\c\lib\gcc\i686-w64-mingw32\4.9.2
libs=-lmoldname -lkernel32 -luser32 -lgdi32 -lwinspool -lcomdlg32
-ladvapi32 -lshell32 -lole32 -loleaut32 -lnetapi32 -luuid -lws2_32
-lmpr -lwinmm -lversion -lodbc32 -lodbccp32 -lcomctl32
perllibs=-lmoldname -lkernel32 -luser32 -lgdi32 -lwinspool
-lcomdlg32 -ladvapi32 -lshell32 -lole32 -loleaut32 -lnetapi32 -luuid
-lws2_32 -lmpr -lwinmm -lversion -lodbc32 -lodbccp32 -lcomctl32
libc=, so=dll, useshrplib=true, libperl=libperl522.a
gnulibc_version=''
dlsrc=dl_win32.xs, dlext=xs.dll, d_dlsymun=undef, ccdlflags=' '
cccdlflags=' ', lddlflags='-mdll -s -L"C:\STRAWB~1\perl\lib\CORE"
-L"C:\STRAWB~1\c\lib"'
---
C:/Strawberry/perl/site/lib/MSWin32-x86-multi-thread-64int
C:/Strawberry/perl/site/lib
C:/Strawberry/perl/vendor/lib
C:/Strawberry/perl/lib
.
---
HOME (unset)
LANG (unset)
LANGUAGE (unset)
LD_LIBRARY_PATH (unset)
LOGDIR (unset)
PATH=C:\Python27\;C:\Python27\Scripts;C:\Program Files
(x86)\Common Files\Intel\Shared
Files\cpp\bin\Intel64;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Strawberry\c\bin;C:\Strawberry\perl\site\bin;C:\Strawberry\perl\bin;C:\ffmpeg\bin;C:\Program
Files (x86)\Git\cmd;C:\Program Files (x86)\NVIDIA
Corporation\PhysX\Common;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;E:\Perl\source;C:\Program
Files (x86)\EaseUS\Todo Backup\bin\x64\
PERL_BADLANG (unset)
SHELL (unset)

[Chas. Owens]

I think the following patch addresses the confusion of what to use
instead of bytes when you feel the need to access the byte level of a
string.

diff --git a/lib/bytes.pm b/lib/bytes.pm
index 6dad41a..77d849d 100644
--- a/lib/bytes.pm
+++ b/lib/bytes.pm
@@ -35,15 +35,24 @@ bytes - Perl pragma to force byte semantics rather
than character semantics

=head1 NOTICE

-This pragma reflects early attempts to incorporate Unicode into perl and
-has since been superseded. It breaks encapsulation (i.e. it exposes the
-innards of how the perl executable currently happens to store a string),
-and use of this module for anything other than debugging purposes is
-strongly discouraged. If you feel that the functions here within might be
-useful for your application, this possibly indicates a mismatch between
-your mental model of Perl Unicode and the current reality. In that case,
-you may wish to read some of the perl Unicode documentation:
-L<perluniintro>, L<perlunitut>, L<perlunifaq> and L<perlunicode>.
+This pragma reflects early attempts to incorporate Unicode into perl and has
+since been superseded by explicit (rather than this pragma's implict) encoding
+using the L<Encode> module:
+
+ use Encode qw/encode/;
+
+ my $utf8_byte_string = encode "UTF-8", $string;
+ my $latin1_byte_string = encode "Latin1", $string;
+
+Because this module breaks encapsulation (i.e. it exposes the innards of how
+the perl executable currently happens to store a string), the byte values that
+result are in an unspecified encoding. Use of this module for anything other
+than debugging purposes is strongly discouraged. If you feel that the
+functions here within might be useful for your application, this possibly
+indicates a mismatch between your mental model of Perl Unicode and the current
+reality. In that case, you may wish to read some of the perl Unicode
+documentation: L<perluniintro>, L<perlunitut>, L<perlunifaq> and
+L<perlunicode>.

=head1 SYNOPSIS




Chas. Owens
http://github.com/cowens
The most important skill a programmer can have is the ability to read.

[Tony Cook via RT]

Post by Chas. Owens
I think the following patch addresses the confusion of what to use
instead of bytes when you feel the need to access the byte level of a
string.
diff --git a/lib/bytes.pm b/lib/bytes.pm
index 6dad41a..77d849d 100644
--- a/lib/bytes.pm
+++ b/lib/bytes.pm
@@ -35,15 +35,24 @@ bytes - Perl pragma to force byte semantics rather
than character semantics
=head1 NOTICE
-This pragma reflects early attempts to incorporate Unicode into perl and
-has since been superseded. It breaks encapsulation (i.e. it exposes the
-innards of how the perl executable currently happens to store a string),
-and use of this module for anything other than debugging purposes is
-strongly discouraged. If you feel that the functions here within might be
-useful for your application, this possibly indicates a mismatch between
-your mental model of Perl Unicode and the current reality. In that case,
-L<perluniintro>, L<perlunitut>, L<perlunifaq> and L<perlunicode>.
+This pragma reflects early attempts to incorporate Unicode into perl and has
+since been superseded by explicit (rather than this pragma's implict) encoding
+
+ use Encode qw/encode/;
+
+ my $utf8_byte_string = encode "UTF-8", $string;
+ my $latin1_byte_string = encode "Latin1", $string;
+
+Because this module breaks encapsulation (i.e. it exposes the innards of how
+the perl executable currently happens to store a string), the byte values that
+result are in an unspecified encoding. Use of this module for anything other
+than debugging purposes is strongly discouraged. If you feel that the
+functions here within might be useful for your application, this possibly
+indicates a mismatch between your mental model of Perl Unicode and the current
+reality. In that case, you may wish to read some of the perl Unicode
+documentation: L<perluniintro>, L<perlunitut>, L<perlunifaq> and
+L<perlunicode>.
=head1 SYNOPSIS

It seems like an improvement to me.

Should it mention utf8::encode()?

Tony

---
via perlbug: queue: perl5 status: open
https://rt.perl.org/Ticket/Display.html?id=125619

[Chas. Owens]

Post by Tony Cook via RT

Post by Chas. Owens
I think the following patch addresses the confusion of what to use
instead of bytes when you feel the need to access the byte level of a
string.
diff --git a/lib/bytes.pm b/lib/bytes.pm
index 6dad41a..77d849d 100644
--- a/lib/bytes.pm
+++ b/lib/bytes.pm
@@ -35,15 +35,24 @@ bytes - Perl pragma to force byte semantics rather
than character semantics
=head1 NOTICE
-This pragma reflects early attempts to incorporate Unicode into perl and
-has since been superseded. It breaks encapsulation (i.e. it exposes the
-innards of how the perl executable currently happens to store a string),
-and use of this module for anything other than debugging purposes is
-strongly discouraged. If you feel that the functions here within might be
-useful for your application, this possibly indicates a mismatch between
-your mental model of Perl Unicode and the current reality. In that case,
-L<perluniintro>, L<perlunitut>, L<perlunifaq> and L<perlunicode>.
+This pragma reflects early attempts to incorporate Unicode into perl and has
+since been superseded by explicit (rather than this pragma's implict) encoding
+
+ use Encode qw/encode/;
+
+ my $utf8_byte_string = encode "UTF-8", $string;
+ my $latin1_byte_string = encode "Latin1", $string;
+
+Because this module breaks encapsulation (i.e. it exposes the innards of how
+the perl executable currently happens to store a string), the byte values that
+result are in an unspecified encoding. Use of this module for anything other
+than debugging purposes is strongly discouraged. If you feel that the
+functions here within might be useful for your application, this possibly
+indicates a mismatch between your mental model of Perl Unicode and the current
+reality. In that case, you may wish to read some of the perl Unicode
+documentation: L<perluniintro>, L<perlunitut>, L<perlunifaq> and
+L<perlunicode>.
=head1 SYNOPSIS

It seems like an improvement to me.
Should it mention utf8::encode()?
Tony
---
via perlbug: queue: perl5 status: open
https://rt.perl.org/Ticket/Display.html?id=125619

At first, I was going to say no, but then I benchmarked them: utf8::encode
is an order of magnitude faster even with the assignment needed to make it
work like Encode::encode. Even factoring out the call to find_encoding
leaves utf::encode twice as fast as $obj->encode(). It also handles
malformed UTF-8 like the "UTF8" encoding (ie it doesn't replace malformed
characters with the replacement character). Here is a second attempt at a
patch.

[Karl Williamson]

Post by Chas. Owens
I think the following patch addresses the confusion of what to use
instead of bytes when you feel the need to access the byte level of a
string.
diff --git a/lib/bytes.pm <http://bytes.pm> b/lib/bytes.pm

<http://bytes.pm>

Post by Chas. Owens
index 6dad41a..77d849d 100644
--- a/lib/bytes.pm <http://bytes.pm>
+++ b/lib/bytes.pm <http://bytes.pm>
@@ -35,15 +35,24 @@ bytes - Perl pragma to force byte semantics

rather

Post by Chas. Owens
than character semantics
=head1 NOTICE
-This pragma reflects early attempts to incorporate Unicode into perl and
-has since been superseded. It breaks encapsulation (i.e. it exposes the
-innards of how the perl executable currently happens to store a string),
-and use of this module for anything other than debugging purposes is
-strongly discouraged. If you feel that the functions here within might be
-useful for your application, this possibly indicates a mismatch between
-your mental model of Perl Unicode and the current reality. In that case,
-L<perluniintro>, L<perlunitut>, L<perlunifaq> and L<perlunicode>.
+This pragma reflects early attempts to incorporate Unicode into perl and has
+since been superseded by explicit (rather than this pragma's

implict)

Post by Chas. Owens
encoding
+
+ use Encode qw/encode/;
+
+ my $utf8_byte_string = encode "UTF-8", $string;
+ my $latin1_byte_string = encode "Latin1", $string;
+
+Because this module breaks encapsulation (i.e. it exposes the

innards

Post by Chas. Owens
of how
+the perl executable currently happens to store a string), the byte
values that
+result are in an unspecified encoding. Use of this module for
anything other
+than debugging purposes is strongly discouraged. If you feel that the
+functions here within might be useful for your application, this possibly
+indicates a mismatch between your mental model of Perl Unicode and
the current
+reality. In that case, you may wish to read some of the perl Unicode
+documentation: L<perluniintro>, L<perlunitut>, L<perlunifaq> and
+L<perlunicode>.
=head1 SYNOPSIS

It seems like an improvement to me.
Should it mention utf8::encode()?
Tony
---
via perlbug: queue: perl5 status: open
https://rt.perl.org/Ticket/Display.html?id=125619
utf8::encode is an order of magnitude faster even with the assignment
needed to make it work like Encode::encode. Even factoring out the call
to find_encoding leaves utf::encode twice as fast as $obj->encode(). It
also handles malformed UTF-8 like the "UTF8" encoding (ie it doesn't
replace malformed characters with the replacement character). Here is a
second attempt at a patch.

Consider doing an approach of doing something like this:

=head1 NAME

bytes - Perl pragma to access the individual bytes of characters

stressing that this is to be mostly confined to temporary debugging
code. And then later say that this pragma used to be for more things,
but don't do that anymore, as it has been found to be broken. I think
that text should incorporate Chas.' patch.

[Aristotle Pagaltzis]

Post by Rob Dixon (via RT)
I recently read "perldoc bytes" and saw
This pragma reflects early attempts to incorporate Unicode into
perl (sic.) and has since been superseded
I think it is a mistake to fail to say /what/ has superseded it, and
the perluni* pods aren't something I should expect to look at if I'm
not using Unicode

It has been superseded by nothing. I/O is in terms of bytes by default.
If you are not using Unicode, you do not need to do anything special.

If, however, you are using code that does at some point decode your
bytes into text, then you have to re-encode it appropriately yourself;
you never could just say `use bytes` and make its decodedness magically
go away.

Post by Rob Dixon (via RT)
In any statement that deprecates "bytes", I think something should
be said about "utf8", which seems to be its dual but is not

Please explain. FWIW, `use utf8` has a *completely* different purpose
than `use bytes`.

Post by Rob Dixon (via RT)
In short, I believe the perluni* should have been
"perlcharacterencoding", as there is no "perlascii" or "perllatin1"
The journey into the promised land of Unicode has been arduous
enough as it is, and the perluni* pods are a major achievement, so I
don't really expect all of that to be ripped apart on my whim. But
perhaps we could do with something like my "perlcharacterencoding"
as an entry point to all of the above?

Somehow I don’t think the documentation is going to get less confusing
if we keep adding more documents to it…

Post by Rob Dixon (via RT)
As you can tell, these are infant musings without any coherent plan.
But I have recently been asked how to enable byte semantics on a
Perl input stream, and I was troubled to find that I was neither
certain nor able to locate the documentation that told me. I suspect
the answer is that file handles have byte semantics by default

Correct.

Post by Rob Dixon (via RT)
and can be opened that way explicitly with a mode of :raw or using
binmode

That is necessary on Windows only because the :crlf layer is added to
filehandles by default there. (And maybe other platforms, but none of
them are really relevant in practice.)

It is also necessary if you want to turn off decoding on a filehandle
after the fact, but if you ask me, you should change your code to avoid
the need to do that – the semantics of doing it are too murky.

Post by Rob Dixon (via RT)
but I am far from sure
Or perhaps I am just asking for a better index into the perldoc tomes
Thank you for reading

Well. The documentation is sprawling and has no coherent organisation,
basically because it has grown by the same principle as is behind your
proposal: someone needed (it) to explain a particular topic, so they
wrote a document or two to explain that topic. But coherent docs do not
happen by that method.

What this produces is documentation that is great so long as you read
each document by itself top to bottom, and you take the time out to sit
down and read every last document, one by one.

If the documentation is supposed to be more penetrable then it must have
some topical organisation, which requires editing by someone with a good
idea of what topics are covered where, who can then decide how some new
topic should be broken up to fit into the existing structure and/or how
the structure needs to be shifted around to make room for the new topic.

In short, you need an architect.

We don’t have one. In fact we are probably even further away from having
one for the docs than we are from one for the interpreter.

Regards,