Repositories encodings are broken (Bug #433)

$ python
>>> s = "japanese" 
>>> u = s.decode("utf-8").encode("utf-16")
>>> u
'\xff\xfej\x00a\x00p\x00a\x00n\x00e\x00s\x00e\x00'

$ python
>>> s = "日本語" 
>>> u = s.decode("utf-8").encode("utf-16")
>>> u
'\xff\xfe\xe5e,g\x9e\x8a'

to_utf8_for_attachments() at source:app/helpers/attachments_helper.rb@13de4cc4#L31 returns "e,g" in "日本語" of UTF-16.

to_utf8() at source:app/helpers/repositories_helper.rb@2e1bcb2a#L119 converts UTF-16 to UTF-8.

$ irb
ruby-1.8.7-p330 :001 > require 'iconv'
 => true 
ruby-1.8.7-p330 :002 > Iconv.conv("UTF-16","UTF-8","日本語")
 => "\377\376\345e,g\236\212" 
ruby-1.8.7-p330 :003 > Iconv.conv("UTF-16","UTF-8","japanese")
 => "\377\376j\000a\000p\000a\000n\000e\000s\000e\000" 

Did you run and confirm filesystem test repository?

Your recent revisions are about ChangeSets.
Filesystem repository does not have ChangeSets.

Filesystem repository reads file directly for equivalent "svn/hg/git cat".

Do you really understand character encoding and internal ChiliProject/Redmine?

http://ci.chiliproject.org/job/chiliproject_unstable/38/DATABASE=postgresql,RUBY=ruby-1.8.6-p420/testReport/%28root%29/RepositoriesFilesystemControllerTest/test_show_utf16/

This includes \000j\000a\000p\000a\000n\000e\000s\000e\000\r\000\n

I changed this lines as following.
source:test/functional/repositories_filesystem_controller_test.rb@a428fa60#L81

japanese -> english

    def test_show_utf16
      with_settings :repositories_encodings => 'UTF-16' do
        get :entry, :id => PRJ_ID, :path => ['japanese', 'utf-16.txt']
        assert_response :success

        assert_select "tr" do
          assert_select "th.line-num" do
            assert_select "a", :text => /2/
          end
          assert_select "td", :content => /english/
        end

      end

But, test passes.

Your test is wrong.

You don't understand 'UTF-16'.

Eric Davis wrote:

4. Please do not reopen issues without contributing any more information. This is the second time today I've had to say this today. https://www.chiliproject.org/issues/440#note-21

I give you more information in English.
You don't understand my information.

Eric Davis wrote:

Do you understand 'UTF-16'?

3. Yes

You don't understand UTF-16.

$ rake test:scm:setup:filesystem
./filesystem_repository/
./filesystem_repository/test
./filesystem_repository/japanese/
./filesystem_repository/japanese/utf-16.txt
./filesystem_repository/japanese/euc-jp.txt
./filesystem_repository/japanese/big-file.txt
./filesystem_repository/dir/
./filesystem_repository/dir/dirfile
./filesystem_repository/dir/subdir/

$ export LANG=en_US.UTF-8
$ locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8" 
LC_NUMERIC="en_US.UTF-8" 
LC_TIME="en_US.UTF-8" 
LC_COLLATE="en_US.UTF-8" 
LC_MONETARY="en_US.UTF-8" 
LC_MESSAGES="en_US.UTF-8" 
LC_PAPER="en_US.UTF-8" 
LC_NAME="en_US.UTF-8" 
LC_ADDRESS="en_US.UTF-8" 
LC_TELEPHONE="en_US.UTF-8" 
LC_MEASUREMENT="en_US.UTF-8" 
LC_IDENTIFICATION="en_US.UTF-8" 
LC_ALL=

$ cat tmp/test/filesystem_repository/japanese/utf-16.txt  \
    | iconv -f utf-16 -t utf-8 | cat -n
     1    UTF-16
     2    japanese
     3    日本語
     4    にほんご
     5    ニホンゴ
     6    日本語
     7    にほんご
     8    ニホンゴ
     9    日本語
    10    にほんご
    11    ニホンゴ
    12    

$ cat tmp/test/filesystem_repository/japanese/utf-16.txt  | od -a
0000000 del   ~   U nul   T nul   F nul   - nul   1 nul   6 nul  cr nul
0000020  nl nul   j nul   a nul   p nul   a nul   n nul   e nul   s nul
0000040   e nul  cr nul  nl nul   e   e   ,   g  rs  nl  cr nul  nl nul
0000060   k   0   {   0 dc3   0   T   0  cr nul  nl nul   K   0   [   0
0000100   s   0   4   0  cr nul  nl nul   e   e   ,   g  rs  nl  cr nul
0000120  nl nul   k   0   {   0 dc3   0   T   0  cr nul  nl nul   K   0
0000140   [   0   s   0   4   0  cr nul  nl nul   e   e   ,   g  rs  nl
0000160  cr nul  nl nul   k   0   {   0 dc3   0   T   0  cr nul  nl nul
0000200   K   0   [   0   s   0   4   0  cr nul  nl nul  cr nul  nl nul
0000220

Eric Davis wrote:

2. 9baa310486d782b has nothing to do with this. Both assert_tag and assert_select are testing the same thing.

Your test is wrong.

This test passes.

    def test_show_eucjp
      with_settings :repositories_encodings => 'UTF-8' do
        get :entry, :id => PRJ_ID, :path => ['japanese', 'euc-jp.txt']
        assert_response :success
        assert_select "td", {:count=>1, :text=>"japanese"}
      end
    end

But, this test fails.

    def test_show_utf16
      with_settings :repositories_encodings => 'UTF-16' do
        get :entry, :id => PRJ_ID, :path => ['japanese', 'utf-16.txt']
        assert_response :success
        assert_select "td", {:count=>1, :text=>"japanese"}
      end
    end

  1) Failure:
test_show_utf16(RepositoriesFilesystemControllerTest)
    [test/functional/repositories_filesystem_controller_test.rb:93:in `test_show_utf16'
     test/test_helper.rb:89:in `with_settings'
     test/functional/repositories_filesystem_controller_test.rb:90:in `test_show_utf16']:
<"japanese"> expected but was
<"U\000T\000F\000-\0001\0006\000\r\000">.
<false> is not true.

UTF-16 is not compatible with ASCII.
EUC-JP and UTF-8 are compatible with ASCII.

You don't understand UTF-16.

@Toshi: It's quite over the top imho to (re-)open THREE times a ticket despite having been asked not to do so, resp. having been asked for more information (which 'You should backout 9baa3104.' certainly is not). Please try not to clutter issues with updates and finally, writing "Your test is wrong. You don't understand 'UTF-16'." is very unfriendly and really not helpful. - thank you

Hans-Peter Suter wrote:

Please try not to clutter issues with updates and finally, writing "Your test is wrong. You don't understand 'UTF-16'." is very unfriendly and really not helpful. - thank you

It is easy to know why RepositoriesFilesystemControllerTest.test_show_utf16 fails.

http://ci.chiliproject.org/job/chiliproject_unstable/DATABASE=mysql,RUBY=ruby-1.8.7-p334/12/testReport/junit/(root)/RepositoriesFilesystemControllerTest/test_show_utf16/

regression