473,320 Members | 2,088 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,320 software developers and data experts.

Shift-JIS to UTF-8 conversion

Hello everybody,

I need to convert a Japanese Shift-JIS CSV file to Unicode UTF-8.

My machine is a Windows 98 english computer with Python 2.3.4

Any hints?.

Jul 19 '05 #1
4 8054
I think you do something like this (untested):

import codecs

def transcode(infile, outfile, incoding="shift-jis",
outcoding="utf-8"):
f = codecs.open(infile, "rb", incoding)
g = codecs.open(outfile, "wb", outcoding)

g.write(f.read())
# If the file is so large that it can't be read at once, do a loop which
# reads and writes smaller chunks
# while 1:
# block = f.read(4096000)
# if not block: break
# g.write(block)

f.close()
g.close()

Jeff

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (GNU/Linux)

iD8DBQFCjRzZJd01MZaTXX0RAg8YAJ4rQ8Fcpwi1AB2a/ZVdALGysct8jACfYdXm
in2aJ3xmdB0ncRZBWXmfMQs=
=bHjV
-----END PGP SIGNATURE-----

Jul 19 '05 #2
Hello,
I think the answer is basically correct but shift-jis is not a standard
part of
Python 2.3. You will either need to use Python 2.4 where the cjkcodes
are integrated or install them under Python 2.3. The link is
http://cjkpython.i18n.org/

You then also need:
import cjkcodecs.aliases

Richard

Jeff Epler wrote:
I think you do something like this (untested):

import codecs

def transcode(infile, outfile, incoding="shift-jis",
outcoding="utf-8"):
f = codecs.open(infile, "rb", incoding)
g = codecs.open(outfile, "wb", outcoding)

g.write(f.read())
# If the file is so large that it can't be read at once, do a loop which # reads and writes smaller chunks
# while 1:
# block = f.read(4096000)
# if not block: break
# g.write(block)

f.close()
g.close()

Jeff


Jul 19 '05 #3
PyTJ wrote:
I need to convert a Japanese Shift-JIS CSV file to Unicode UTF-8.

My machine is a Windows 98 english computer with Python 2.3.4

Any hints?.


First, you need to install codecs to support japanese encodings.
Python 2.3.* does not support SJIS by default.

I'll give you two options.

- Japanese Codecs
http://www.python.jp/Zope/download/JapaneseCodecs

http://ftp.python.jp/pub/JapaneseCod...in32-py2.3.exe

- CJKCodecs
http://cjkpython.i18n.org/
http://download.berlios.de/cjkpython...in32-py2.3.exe

If you only need Japanese support, Japanese Codecs might be handy.
On the other hand, CJKCodecs can handle much broader encodings.
Aside from that, starting from 2.4, Python ships with CJKCodecs,
so I'd recomment CJKCodecs without reservations.

-- george
Jul 19 '05 #4
On Fri, May 20, 2005 at 12:16:15AM -0700, rb*****@gmx.de wrote:
Hello, I think the answer is basically correct but shift-jis is not a
standard part of Python 2.3.


Ah, I was fooled --- I tested on Python 2.3, but my packager must have
included the codecs you went on to mention.

Jeff

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.1 (GNU/Linux)

iD8DBQFCkcJCJd01MZaTXX0RAhNUAKCLbSsAAzxXe9UIjMXd5A N/wKcfbQCeI9j0
lpU5Zu0BgAdD2hTFvKB8kJs=
=Tof0
-----END PGP SIGNATURE-----

Jul 19 '05 #5

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

5
by: Alex Stevens | last post by:
Hi All, I'm writing a launcher program which checks for updates and copies them down to the client. However, I would like the user to be able to modify the settings for the launcher. As this...
6
by: PerryC | last post by:
I have search googles and there are hundreds of tips about AllowByPassKey... however, none works for me... well, perhaps I am too new to such high level functionality that it just does not make...
388
by: maniac | last post by:
Hey guys, I'm new here, just a simple question. I'm learning to Program in C, and I was recommended a book called, "Mastering C Pointers", just asking if any of you have read it, and if it's...
43
by: Mehta Shailendrakumar | last post by:
Hello, Can anyone suggest me operator to perform arithmetic shift in C? May it be for a perticular compiler. Thank you in advance. Regards, Shailendra
10
by: Tim Frawley | last post by:
I am attempting to detect a Shift+Tab in the KeyPress event for back navigation on a control that doesn't support this method. Does anyone have any ideas how to compare e.KeyChar to a ShiftTab? ...
7
by: Csaba Gabor | last post by:
I'd like to detect the shift key when a button is "clicked" in Firefox/Mozilla. If the button is clicked with the mouse, no problem. However, if the onclick event is keyboard originated, then my...
9
by: deepak | last post by:
If i'm shifting an integer 'n' times where n > sizeof(int), It's giving the same value as that of n-sizeof(n). Why is it behaving like this. I studied that the bit 'll be replaced by '0'. ...
4
by: sandhya | last post by:
Hello Folks, i hava a problem in coding of circular left shift of 25 bits in my program...how do i perform it, and how do i use unsigned in VB. My program (IDEA algorithm implementation in VB) ...
3
by: Amy Smith | last post by:
Hello there, I am having a small problem which been challenging me for few days and need help or advice. I am trying to calculate the day-shift for employees based on the time they started and...
3
by: Pietro | last post by:
Hi all, First of all I'd like to thank you very very much ,as finally after many years of searching,I could find a code to disable/enable the shift key,but actually i cannot use the code as I'm...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
1
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome former...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.