Hey guys! In this tutorial, we will learn about Unicode in Python and the character properties of Unicode. So, let’s get started.<\/p>\n\n\n\n
Unicode associates each character and symbol with a unique number called code points. It supports all of the world’s writing systems and ensures that data can be retrieved or combined using any combination of languages.<\/p>\n\n\n\n
The codepoint is an integer value ranging from 0 to 0x10FFFF in hexadecimal coding.<\/p>\n\n\n\n
To begin using Unicode characters in Python, we need to understand how the string module interprets characters. <\/p>\n\n\n\n
Python provides us a string<\/em> module that contains various functions and tools to manipulate strings. It falls under the ASCII character set.<\/p>\n\n\n Output:<\/p>\n\n\n We can create one-character Unicode strings by using chr() <\/strong>built-in function. It takes only one integer as argument and returns the unicode of the given character. <\/p>\n\n\n\n Similarly, odr() is an inbuilt function that takes a one-character Unicode string as input and returns the code point value. <\/p>\n\n\n Output:<\/p>\n\n\n A string is a sequence of Unicode codepoints. These codepoints are converted into a sequence of bytes for efficient storage. This process is called character encoding.<\/p>\n\n\n\n There are many encodings such as UTF-8,UTF-16,ASCII etc. <\/p>\n\n\n\n By default, Python uses UTF-8 encoding.<\/strong><\/p>\n\n\n\n UTF-8 is the most popular and commonly used for encoding characters. UTF stands for Unicode Transformation Format<\/em> and ‘8’ means that 8-bit values<\/em> are used in the encoding. <\/p>\n\n\n\n It replaced ASCII (American Standard Code For Information Exchange) as it provides more characters and can be used for different languages around the world, unlike ASCII which is only limited to Latin languages. <\/p>\n\n\n\n The first 128 codepoints in the UTF-8 character set are also valid ASCII characters. A character in UTF-8 can be from 1 to 4 bytes long.<\/p>\n\n\n\n The encode() method<\/a> converts any character from one encoding to another. The syntax of the encode function is as shown below –<\/p>\n\n\n Parameters<\/strong>:<\/p>\n\n\n\n Let’s now move to understanding how the string encode function can allow us to create unicode strings in Python. <\/p>\n\n\n\n Output:<\/strong><\/p>\n\n\n Let us encode the german word wei\u00df which means white.<\/p>\n\n\n Output:<\/p>\n\n\n The unicodedata<\/strong> <\/em>module provides us the Unicode Character Database (UCD)<\/em><\/strong> which defines all character properties of all Unicode characters.<\/p>\n\n\n\n Let’s look at all the functions defined within the module with a simple example to explain their functionality. We can efficiently use Unicode in Python with the use of the following functions. <\/p>\n\n\n\n This function looks up the character by the given name. If the character is found, the corresponding character is returned. If not found, then Keyerror is raised.<\/p>\n\n\n Output:<\/p>\n\n\n This function returns the name assigned to character chr <\/em>as string. If no name is defined, it returns the default otherwise it raises Keyerror.<\/p>\n\n\n Output:<\/p>\n\n\n This function returns the decimal value assigned to the character chr<\/em>. If no value is defined then the default is returned otherwise Keyerror is raised as shown in the example below.<\/p>\n\n\n Output:<\/p>\n\n\n This function returns the digit value assigned to the character chr <\/em>as integer. One thing to note is that this function takes a single character as an input. In the last line in this example, I’ve used “20” and the function throws an error stating that it cannot accept a string as an input.<\/p>\n\n\n Output:<\/p>\n\n\n This function returns the numeric value assigned to the character chr <\/em>as an integer. If no value is defined then it returns default otherwise ValueError is raised.<\/p>\n\n\n Output:<\/p>\n\n\n This function returns the general category assigned to the character chr <\/em>as a string. It returns \u2018L\u2019 for letter and \u2018u\u2019 for uppercase and \u2018l\u2019 for lowercase.<\/p>\n\n\n Output:<\/p>\n\n\n This function returns the bidirectional class assigned to the character chr as a string. An empty string is returned by this function if no such value is defined. <\/p>\n\n\n\n AL denotes Arabic letter, AN denotes Arabic number and L denotes left to right etc.<\/p>\n\n\n Output:<\/p>\n\n\n This function returns canonical combining class assigned to the given character chr <\/em>as string. It returns 0 if there is no combining class defined.<\/p>\n\n\n Output:<\/p>\n\n\n This function returns a mirrored <\/em>property assigned to the given character chr<\/em> as an integer. It returns 1<\/em> if the character is identified as ‘mirrored<\/em>‘ in bidirectional text or else it returns 0<\/em>.<\/p>\n\n\n Output:<\/p>\n\n\n Using this function returns the conventional form for the Unicode string unistr. The valid values for form are \u2018NFC\u2019, \u2018NFKC\u2019, \u2018NFD\u2019, and \u2018NFKD\u2019.<\/p>\n\n\n Output:<\/p>\n\n\n <\/p>\n\n\n\n In this tutorial, we learned about unicode and unicodedatabase module which defines the unicode characteristics. Hope you all enjoyed. Stay Tuned \ud83d\ude42<\/p>\n\n\n\n\nimport string\n\nprint(string.ascii_lowercase) \nprint(string.ascii_uppercase)\nprint(string.ascii_letters)\nprint(string.digits)\nprint(string.hexdigits)\nprint(string.octdigits)\nprint(string.whitespace) \nprint(string.punctuation)\n<\/pre><\/div>\n\n\n
\nABCDEFGHIJKLMNOPQRSTUVWXYZ\nabcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ\n0123456789\n0123456789abcdefABCDEF\n01234567\n \t\n!"#$%&'()*+,-.\/:;<=>?@[\\]^_`{|}~\n<\/pre><\/div>\n\n\n\nchr(57344)\nord('\\ue000')\n<\/pre><\/div>\n\n\n\n'\\ue000'\n57344\n<\/pre><\/div>\n\n\n
What does character encoding mean in Python?<\/h2>\n\n\n\n
What is UTF-8 Encoding?<\/h2>\n\n\n\n
Encoding Characters in UTF-8 Using the Python encode() function<\/h2>\n\n\n\n
\nstring.encode(encoding='UTF-8',errors='strict')\n<\/pre><\/div>\n\n\n
How to use Unicode in Python with the encode() function?<\/h2>\n\n\n\n
1. Encode a string to UTF-8 encoding<\/h3>\n\n\n
\nstring = '\u00f6range'\nprint('The string is:',string)\nstring_utf=string.encode()\nprint('The encoded string is:',string_utf)\n<\/pre><\/div>\n\n\n\nThe string is: \u00f6range\nThe encoded string is: b'\\xc3\\xb6range'\n<\/pre><\/div>\n\n\n
2. Encoding with error parameter<\/h3>\n\n\n\n
\nstring = 'wei\u00df'\n\nx = string.encode(encoding='ascii',errors='backslashreplace')\nprint(x)\n\nx = string.encode(encoding='ascii',errors='ignore')\nprint(x)\n\nx = string.encode(encoding='ascii',errors='namereplace')\nprint(x)\n\nx = string.encode(encoding='ascii',errors='replace')\nprint(x)\n\nx = string.encode(encoding='ascii',errors='xmlcharrefreplace')\nprint(x)\n\nx = string.encode(encoding='UTF-8',errors='strict')\nprint(x)\n<\/pre><\/div>\n\n\n
\nb'wei\\\\xdf'\nb'wei'\nb'wei\\\\N{LATIN SMALL LETTER SHARP S}'\nb'wei?'\nb'wei\u00df'\nb'wei\\xc3\\x9f'\n\n<\/pre><\/div>\n\n\nThe uncidedata module to work with Unicode in Python<\/h2>\n\n\n\n
1. unicodedata.lookup(name)<\/strong><\/h3>\n\n\n\n
\nimport unicodedata \n \nprint (unicodedata.lookup('LEFT CURLY BRACKET')) \nprint (unicodedata.lookup('RIGHT SQUARE BRACKET')) \nprint (unicodedata.lookup('ASTERISK'))\nprint (unicodedata.lookup('EXCLAMATION MARK'))\n<\/pre><\/div>\n\n\n\n{\n]\n*\n!\n<\/pre><\/div>\n\n\n2. unicodedata.name(chr[, default])<\/strong><\/h3>\n\n\n\n
\nimport unicodedata \n \nprint (unicodedata.name(u'%')) \nprint (unicodedata.name(u'|')) \nprint (unicodedata.name(u'*')) \nprint (unicodedata.name(u'@'))\n\n<\/pre><\/div>\n\n\n
\nPERCENT SIGN\nVERTICAL LINE\nASTERISK\nCOMMERCIAL AT\n<\/pre><\/div>\n\n\n
3. unicodedata.decimal(chr[, default])<\/strong><\/h3>\n\n\n\n
\nimport unicodedata\n \nprint (unicodedata.decimal(u'6'))\nprint (unicodedata.decimal(u'b')) \n<\/pre><\/div>\n\n\n
\n6\nTraceback (most recent call last):\n File "D:\\DSCracker\\DS Cracker\\program.py", line 4, in <module>\n print (unicodedata.decimal(u'b')) \nValueError: not a decimal\n<\/pre><\/div>\n\n\n
4. unicodedata.digit(chr[, default])<\/strong><\/h3>\n\n\n\n
\nimport unicodedata \n \nprint (unicodedata.decimal(u'9')) \nprint (unicodedata.decimal(u'0')) \nprint (unicodedata.decimal(u'20'))\n<\/pre><\/div>\n\n\n
\n9\n0\nTraceback (most recent call last):\n File "D:\\DSCracker\\DS Cracker\\program.py", line 5, in <module>\n print (unicodedata.decimal(u'20'))\nTypeError: decimal() argument 1 must be a unicode character, not str\n<\/pre><\/div>\n\n\n
5. unicodedata.numeric(chr[, default])<\/strong><\/h3>\n\n\n\n
\nimport unicodedata \n \nprint (unicodedata.decimal(u'1'))\nprint (unicodedata.decimal(u'8'))\nprint (unicodedata.decimal(u'123'))\n<\/pre><\/div>\n\n\n
\n1\n8\nTraceback (most recent call last):\n File "D:\\DSCracker\\DS Cracker\\program.py", line 5, in <module>\n print (unicodedata.decimal(u'123')) \nTypeError: decimal() argument 1 must be a unicode character, not str\n<\/pre><\/div>\n\n\n
6. unicodedata.category(chr)<\/strong><\/h3>\n\n\n\n
\nimport unicodedata \n \nprint (unicodedata.category(u'P')) \nprint (unicodedata.category(u'p')) \n<\/pre><\/div>\n\n\n
\nLu\nLl\n<\/pre><\/div>\n\n\n
7. unicodedata.bidirectional(chr)<\/strong><\/h3>\n\n\n\n
\nimport unicodedata \n \nprint (unicodedata.bidirectional(u'\\u0760'))\n\nprint (unicodedata.bidirectional(u'\\u0560')) \n\nprint (unicodedata.bidirectional(u'\\u0660')) \n\n\n<\/pre><\/div>\n\n\n
\nAL\nL\nAN\n<\/pre><\/div>\n\n\n
8. unicodedata.combining(chr)<\/strong><\/h3>\n\n\n\n
\nimport unicodedata \n \nprint (unicodedata.combining(u"\\u0317"))\n\n<\/pre><\/div>\n\n\n
\n220\n<\/pre><\/div>\n\n\n
9. unicodedata.mirrored(chr)<\/h3>\n\n\n\n
\nimport unicodedata \n \nprint (unicodedata.mirrored(u"\\u0028"))\nprint (unicodedata.mirrored(u"\\u0578"))\n\n<\/pre><\/div>\n\n\n
\n1\n0\n<\/pre><\/div>\n\n\n
10. unicodedata.normalize(form, unistr)<\/strong><\/h3>\n\n\n\n
\nfrom unicodedata import normalize \n \nprint ('%r' % normalize('NFD', u'\\u00C6')) \nprint ('%r' % normalize('NFC', u'C\\u0367')) \nprint ('%r' % normalize('NFKD', u'\\u2760')) \n\n<\/pre><\/div>\n\n\n\n'\u00c6'\n'C\u0367'\n'\u2760'\n<\/pre><\/div>\n\n\n
Conclusion<\/h2>\n\n\n\n
References<\/h2>\n\n\n\n