Deep Learning Implementation to Detect and Translation Sign Language Into Alphabet Using Convolutional Neural Network = Implementasi Deep Learning untuk Mendeteksi dan Menterjemahkan Bahasa Isyarat menjadi Alphabet Menggunakan Convolution Neural Network

Tedjasurya, Steven Jonathan (2022) Deep Learning Implementation to Detect and Translation Sign Language Into Alphabet Using Convolutional Neural Network = Implementasi Deep Learning untuk Mendeteksi dan Menterjemahkan Bahasa Isyarat menjadi Alphabet Menggunakan Convolution Neural Network. Bachelor thesis, Universitas Pelita Harapan.

Preview	Text (Title.pdf) Title.pdf.pdf Available under License Creative Commons Attribution Non-commercial Share Alike. Download (42kB) | Preview
Preview	Text (Abstract.pdf) Abstract.pdf.pdf Available under License Creative Commons Attribution Non-commercial Share Alike. Download (97kB) | Preview
Preview	Text (ToC.pdf) ToC.pdf.pdf Available under License Creative Commons Attribution Non-commercial Share Alike. Download (141kB) | Preview
Preview	Text (Chapter1.pdf) Chapter1.pdf.pdf Available under License Creative Commons Attribution Non-commercial Share Alike. Download (81kB) | Preview
	Text (Chapter2.pdf) Chapter2.pdf.pdf Restricted to Registered users only Available under License Creative Commons Attribution Non-commercial Share Alike. Download (560kB)
	Text (Chapter3.pdf) Chapter3.pdf.pdf Restricted to Registered users only Available under License Creative Commons Attribution Non-commercial Share Alike. Download (148kB)
	Text (Chapter4.pdf) Chapter4.pdf.pdf Restricted to Registered users only Available under License Creative Commons Attribution Non-commercial Share Alike. Download (464kB)
	Text (Chapter5.pdf) Chapter5.pdf.pdf Restricted to Registered users only Available under License Creative Commons Attribution Non-commercial Share Alike. Download (849kB)
	Text (Chapter6.pdf) Chapter6.pdf.pdf Restricted to Registered users only Available under License Creative Commons Attribution Non-commercial Share Alike. Download (860kB)
Preview	Text (Bibliography) Bibliography.pdf Available under License Creative Commons Attribution Non-commercial Share Alike. Download (150kB) | Preview
	Text (Appendices) Appendices.pdf Restricted to Repository staff only Available under License Creative Commons Attribution Non-commercial Share Alike. Download (3MB)

Abstract

Sign language is one of the languages that lack very far behind in translation technology, unlike ordinary language that is often spoken for a non-disabled person that has very mature technology. Global Burden of Disease Study (GBD) study in 2019 showed that about 1.57 billion people have some form of hearing loss either on permanent or non-permanent condition, 35 million are estimated to have permanent hearing loss. This creates a demand for a reliable digital sign language translator to accommodate a disabled person for easier daily communication. To detect sign language gestures images, a model is trained with a dataset, the sign language MNIST data and the author artificial data, which is trained with convolution neural network architecture with TensorFlow. Models generated are then tested in OpenCV. The researcher produced 3 models for the sign language MNIST dataset, Epochs 10 has an accuracy value of 87.1%, a precision value of 88.60%, a recall value of 87.10%, and an F-1 score value of 87.10%. Epochs 30 has an accuracy of 82.90%, a precision value of 87.60%, a recall value of 82.90%, and an F-1 score value of 83.60%. Epochs 50 has an accuracy value of 86.30 %, a precision value of 88.70%, a recall value of 86.30%, and an F-1 score value of 86.30%. For the author dataset models, the researcher produced epoch 50, and 100. Epochs 50 has an accuracy value of 81.6%, a precision value of 85.80%, a recall value of 81.60%, and an F-1 score value of 79.70%. Epochs 100 has an accuracy value of 92.9%, a precision value of 94%, a recall value of 94%, and an F-1 value of 94%. The generated model of the training is run on a software called OpenCV, based on the result from the OpenCV, epochs 10 managed to detect 16/24 alphabets, epoch 30 managed to detect 14/24 alphabets, while epoch 50 managed to detect 14/24 alphabets. When interference is given during the detection test, epochs 10 only managed to detect 6/24 alphabets, epochs 30 only managed to detect 3/24 alphabets, while epoch 50 managed to detect 3/24 alphabets. While the author dataset for epochs 50 and 100 displays bad results in detection test if compare with the sign language MNIST trained model dataset tests. / Bahasa isyarat merupakan salah satu bahasa yang sangat tertinggal jauh dalam teknologi penerjemahan, tidak seperti bahasa biasa yang sering diucapkan oleh orang non-disabilitas yang teknologinya sudah sangat matang. Studi dari Global Burden of Disease Study (GBD) pada tahun 2019, menunjukkan bahwa sekitar 1,57 miliar orang memiliki beberapa bentuk gangguan pendengaran baik permanen ataupun non-permanen, 35 juta diperkirakan mengalami gangguan pendengaran permanen. Hal ini menciptakan permintaan akan penerjemah bahasa isyarat digital yang andal untuk mengakomodasi penyandang disabilitas untuk komunikasi sehari-hari yang lebih mudah. Untuk mendeteksi isyarat bahasa isyarat, model dilatih dengan kumpulan data, data MNIST bahasa isyarat dan data buatan penulis dilatih dengan arsitektur jaringan syaraf konvolusi menggunakan TensorFlow. Model yang dihasilkan kemudian diuji menggunakan OpenCV. Peneliti menghasilkan 3 model untuk dataset bahasa isyarat MNIST, Epochs 10 memiliki nilai akurasi 87,1%, nilai presisi 88,60%, nilai recall 87,10%, dan nilai skor F-1 87,10%. Epochs 30 memiliki akurasi 82,90%, nilai presisi 87,60%, nilai recall 82,90%, dan nilai skor F-1 83,60%. Epochs 50 memiliki nilai akurasi 86,30%, nilai presisi 88,70%, nilai recall 86,30%, dan nilai skor F-1 86,30%. Untuk model dataset penulis, peneliti menghasilkan epoch 50, dan 100. Epochs 50 memiliki nilai akurasi 81,6%, nilai presisi 85,80%, nilai recall 81,60%, dan nilai skor F-1 79,70%. Epochs 100 memiliki nilai akurasi 92,9%, nilai presisi 94%, nilai recall 94%, dan nilai F-1 94%. Model pelatihan yang dihasilkan akan dijalankan pada software bernama OpenCV, berdasarkan hasil dari OpenCV, epoch 10 berhasil mendeteksi 16/24 abjad, epoch 30 berhasil mendeteksi 14/24 abjad, sedangkan epoch 50 berhasil mendeteksi 14 /24 huruf. Saat diberikan interferensi saat uji deteksi, epoch 10 hanya berhasil mendeteksi 6/24 abjad, epoch 30 hanya berhasil mendeteksi 3/24 abjad, sedangkan epoch 50 berhasil mendeteksi 3/24 abjad. Sementara dataset penulis untuk epoch 50 dan 100 menampilkan hasil yang buruk dalam tes deteksi, jika dibandingkan dengan tes dataset model terlatih MNIST bahasa isyarat.

Actions (login required)

View Item