Local Attention<\/strong>: In Local attention, some of the hidden states of the LSTM layers are used to generate the context vector.<\/p>\n You must be familiar with Python, Deep Learning, and Natural Language Processing(NLP). Using the following command, you can install all of the project’s modules:<\/p>\n The versions of Python and its relevant modules used in this project are as follows:<\/p>\n You can download the datasets for Twitter Hashtag prediction project from the following link: Twitter Hashtag prediction Dataset<\/strong><\/a><\/p>\n Please download the source code of twitter hashtag prediction project with machine learning: Twitter Hashtag Prediction Project Code<\/strong><\/a><\/p>\n train.csv<\/strong>: This file contains training data for our model. It has tweets, locations, and a confidence score for each of 24 possible labels. These 24 labels are from three categories: sentiment, when and kind.<\/p>\n test.csv<\/strong>: This file contains testing data for our model. It has tweets and locations.<\/p>\n twitter_hashtag.py<\/strong>: In this file, we will create and train our model with input and target to predict the hashtag for the tweet.<\/p>\n s2s\/<\/strong>: Our trained model’s optimizer, metrics, and weights are all stored in this directory.<\/p>\n contractions.pkl<\/strong>: This file provides a dictionary with keys representing abbreviated words and values representing their extended or original meanings.<\/p>\n Firstly we will create a file called \u2018twitter_hashtag.py\u2019 and import all the libraries which have been shared in the prerequisites section.<\/p>\n Code:<\/strong><\/p>\n We will traverse the dataset folder which contains the \u2018train.csv\u2019 file and extract all the input texts. Our input will be the \u2018tweet\u2019 column which contains users’ tweets and the target label will be s1,s2,s3,s4,s5<\/strong> i.e \u201csentiment\u201d, w1,w2,w3,w4<\/strong> i.e \u201cwhen\u201d and k1,k2,k3….k15<\/strong> i.e \u201ckind\u201d column. The target label value will be the hashtag value of respective labels from s1 to k15.<\/p>\n Code:<\/strong><\/p>\n Input Data from \u2018train.csv\u2019<\/strong><\/p>\n Target Data from \u2018train.csv\u2019:<\/strong><\/p>\n Label Value:<\/strong><\/p>\n We need to clean our tweet texts as this data might be incomplete and it cannot be sent directly to the model. So we will create a function \u2018clean()\u2019 which does the following process in order to clean the data and returns the cleaned words: Code:<\/strong><\/p>\n Now we will pass our input data that is training tweets to the \u2018clean()\u2019 function to get clean words and tweets.<\/p>\n Code:<\/strong><\/p>\n As we are using many to many encoder-decoder sequence models, we need to make our target data also ready for the model. We will convert our confidence scores of \u201csentiments\u201d, \u201cwhen\u201d and \u201ckind\u201d data back in the form of labels. As we can take only one value from \u201csentiments\u201d and \u201cwhen\u201d labels, so for that we will take the maximum confidence score. And for the \u201ckind\u201d type we can have as many but we will take values that have a confidence score greater than 0.5.<\/p>\n We also need to have a starting and ending point for the target data as this will tell the model that from the starting word the sentence begins and after reaching the ending word the sentence ends. For example, we will take the following sentence \u201cTechVidvan provides many projects\u201d<\/strong> and now we will add \u201csos\u201d (start of a sentence) at the starting of this sentence and \u201ceos\u201d<\/strong> (end of the sentence) at the end of that sentence. So our new sentence will be \u201csos TechVidvan provides many projects eos\u201d. Let\u2019s see how it works:<\/p>\n Input(x)\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0Output(y)<\/strong> As you can see after feeding the input(x) word we get output(y) which is the next word. After reaching \u201ceos\u201d we will stop as it indicates the end of the sentence.<\/p>\n Code:<\/strong><\/p>\n Now let’s see how many unique words are in input and target data along with the maximum input and target length.<\/p>\n Code:<\/strong><\/p>\n Output:<\/strong><\/p>\n We will split the input and target texts into a 90:10 ratio. 90% for the training set and 10% for the testing set.<\/p>\n Code:<\/strong><\/p>\n As our model requires numeric data, we will use text vectorization. Text Vectorization is a technique that converts textual data into numeric data. Numeric data can be the index of respective textual data.<\/p>\n For example, D={ \u2018on\u2019:1, \u2018monday\u2019:2, \u2018free\u2019:3, \u2018product\u2019:4, \u2018you\u2019:5, \u2018are\u2019:6, \u2018best\u2019:7 }<\/strong> S = \u201care you free on monday\u201d<\/strong><\/p>\n Transformed (Vectorized) S : [ 6 , 5 , 3 , 1 , 2 ]<\/strong> Code:<\/strong><\/p>\n We’ll make all of the input and target texts the same length after converting them to integer sequences for our model. We will pad an array of 0’s to the integer sequence if it is smaller than the provided maximum input length using the ‘max in len’ and ‘max tr len’ variables.<\/p>\n We’ll pad our encoder input data (‘x train’) and our decoder input data (‘y train’) but we won’t include the last word, ‘eos.’ The decoder target data will be the same as the decoder input data, but it will be one timestep ahead since it will not include our target sentence’s first word,’sos.’<\/p>\n Code:<\/strong><\/p>\n The LSTM network will have three stacked layers. This will greatly improve our prediction. As per your requirement, you can have more also. Let’s have a look at our encoder and decoder models.<\/p>\n Encoder<\/span>:<\/strong> The ‘Input’ object will be used to initialize the encoder input tensor. The batch is expected to have 14 (maximum input length) dimensions. Then we’ll make a ‘Embedding Layer,’ with the total number of input words as the first argument and a shape of 500 for the latent (hidden) dimension.<\/p>\n LSTM<\/span>:<\/strong> The input of the encoder will be used in the first LSTM layer, and so on, creating a continuous succession of LSTM layers. All of the contextual information in the input sequence will be captured by the LSTM layer. Following the execution of each LSTM layer, we will deliver hidden state output as well as states (hidden state and cell state).<\/p>\n Code:<\/strong><\/p>\n Decoder<\/span>:<\/strong> We’ll build the decoder input tensor similarly to Encoder and then give it to the single LSTM. The initial state of the decoder is where we will pass the hidden state and cell state values that we obtained from the encoder’s LSTM layer.<\/p>\n Attention Layer<\/span>:<\/strong> The encoder and decoder outputs will be passed to the attention layer, which will then concatenate the attention layer outputs with the decoder outputs.<\/p>\n Code:<\/strong><\/p>\n We’ll now build our Dense Layer, which will be the model’s output layer. It will have the output dimension the same as a number of target word units of output dimension along with the softmax activation function.<\/p>\n Code:<\/strong><\/p>\n Finally, we’ll set up encoder and decoder inputs, as well as decoder output, in our Model class. We’ll plot the model layers and get a summary of our model.<\/p>\n Code:<\/strong><\/p>\n Output:<\/strong><\/p>\n <\/p>\n Now we will train our model by passing the encoder input data, decoder input data as input for the model and decoder target data as output with \u2018500\u2019 batch size, for \u201810\u2019 epochs and the optimizer will be \u2018RMSprop\u2019. You can increase or decrease the epoch but take care of the validation accuracy and loss.<\/p>\n Code:<\/strong><\/p>\n Output:<\/strong><\/p>\n After training the model, we got an accuracy of 95.49%.<\/p>\n We will be using the trained model to create an inference architecture for the encoder and decoder model. Load the model. By using the Inference model we will test our model on new sentences for which the target sequence is unknown.<\/p>\n Code:<\/strong><\/p>\n Encoder Inference:<\/strong> <\/span>The 0th layer, i.e. the input object that we constructed (as seen in the above summary and model plot), will be the input for the inference encoder model, and the output will be the output of the last LSTM, which is the 6th layer.<\/p>\n Code:<\/strong><\/p>\n Decoder Inference:<\/strong><\/span> We’ll get the input, embedding, and LSTM layers from the stored model, just as the Encoder inference model. The shape of latent (hidden) dimensions should be used to initialize the decoder hidden input and the other two states.<\/p>\n Attention Inference:<\/strong> <\/span>The attention layer is the eighth layer in our case. We’ll get it and combine the output of the inference decoder with the hidden state input we set up earlier. The attention layer output will then be concatenated with the decoder output.<\/p>\n Code:<\/strong><\/p>\n And same for the Dense layer (output layer) which is the 10th layer of our saved model. With the help of the above data, initialize the Inference Model class.<\/p>\n Code:<\/strong><\/p>\n Assign state vectors to the input sequence. For each pair, create an empty array containing the intended sequence and generate the start word, which in our case is \u2018sos\u2019. Predict the output index using this state value and the input sequence. To obtain the word from the output index, use the reverse target word index and append it to the decoded sequence.<\/p>\n Code:<\/strong><\/p>\n Assign the index of our word to the target sequence, so that our target sequence has a vector of the previous word for the next iteration. Iterate until our word equals the last word (in our example, ‘eos’) or the target text’s maximum length.<\/p>\n Code:<\/strong><\/p>\n Finally, we are done with all the processes and now we can predict the Hashtag for the input tweet. For this, we will clean and vectorize our \u2018x_test\u2019 and then pass to the \u2018decode_sequence()\u2019 function to get the predicted hashtag.<\/p>\n Code:<\/strong><\/p>\nEncoder-Decoder model:<\/h3>\n
![\"encoder](\"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/sites\/2\/2021\/12\/encoder-decoder-architecture.webp\")
<\/a><\/p>\nPrerequisites :<\/h3>\n
pip install numpy, pandas, pickle, nltk, tensorflow, sklearn<\/pre>\n
\n
Datasets for Twitter Hashtag Prediction Project:<\/h3>\n
Download Twitter Hashtag Prediction Project Code<\/h3>\n
Project Structure :<\/h3>\n
Steps for Twitter Hashtag Prediction ML Project:<\/h3>\n
1) Import the Libraries<\/h4>\n
#TechVidvan Twitter Hashtag Prediction Project\n#import all the required libraries\nimport numpy as np\nimport pandas as pd\nimport pickle\nfrom statistics import mode\nimport nltk\nfrom nltk import word_tokenize\nfrom tensorflow.keras.models import Model\nfrom tensorflow.keras import models\nfrom tensorflow.keras.preprocessing.sequence import pad_sequences\nfrom tensorflow.keras.preprocessing.text import Tokenizer\nfrom tensorflow.keras.utils import plot_model\nfrom tensorflow.keras.layers import Input,LSTM,Embedding,Dense,Concatenate,Attention\nfrom sklearn.model_selection import train_test_split\nnltk.download('wordnet')\nnltk.download('punkt')<\/pre>\n2) Parse the Dataset file.<\/h4>\n
#read the dataset file\ntrain=pd.read_csv(\"train.csv\")\n\n#tweet column is input\ninp_data=train[\"tweet\"]\n#target data is sentiment(s1,s2,s3,s4,s5) ,\n#when (w1,w2,w3,w4) and kind(k1,k2,k3...k15)\ntar_data=train.iloc[:,4:].values\n\n#get the column name of target\ntar_lab=train.iloc[:,4:].columns.tolist()\n\n#value of the target label like\n#s1=\"I can't tell\" , s2=\"Negative\" and so on till s5\n#w1=\"current weather\", w2=future forecast and so on till w4\n#k1=\"clouds\", k2=\"cold\", k3=\"dry\" and so on till k15\ntar_lab_val=[\n\"I can't tell\",\"Negative\",\"Neutral\",\"Positive\",\"Tweet not related to weather condition\",\n\"current (same day) weather\",\"future (forecast)\",\"I can't tell\",\"past weather\",\n\"clouds\",\"cold\",\"dry\",\"hot\",\"humid\",\"hurricane\",\"I can't tell\",\"ice\",\"other\",\"rain\",\n\"snow\",\"storms\",\"sun\",\"tornado\",\"wind\"]<\/pre>\n
<\/a><\/p>\n<\/a><\/p>\n<\/a><\/p>\n3) Preprocessing<\/h4>\n
\na) Remove numbers, Alphanumeric words i.e. words which contain both alphabets and integers e.g. hello123 .
\nb) Expand the contraction words which are combinations of two words, shortened using apostrophe for example \u2018aren\u2019t\u2019 shortened for \u2018are not\u2019.<\/p>\n#clean the tweets\ndef clean(tweet):\n #replace and lower case the tweets\n tweet=tweet.replace(\":\",\"\").lower()\n #get only words that contains alphabets\n words= list(filter(lambda w:(w.isalpha()),tweet.split(\" \")))\n #expand the shortened words\n words= [contractions[w] if w in contractions else w for w in words ]\n #return all the words\n return words<\/pre>\n
#iterate over input data\nfor tweet in inp_data:\n #clean the tweets\n inpt_words= clean(tweet)\n #store the input texts and words\n inp_texts+= [' '.join(inpt_words)]\n inp_words+= inpt_words<\/pre>\n
\nsos<\/strong>,\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 TechVidvan
\nsos TechVidvan,\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 provides
\nsos TechVidvan provides,\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0many
\nsos TechVidvan provides many,\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 projects
\nsos TechVidvan provides many projects,\u00a0 \u00a0 \u00a0 eos<\/strong><\/p>\n#iterate over target data\nfor lab in tar_data:\n #get index of maximum value from sentiment data(s1 to s5)\n #with the help of this index get label value\n senti=tar_lab[np.argmax(lab[:5])]\n\n #get index of maximum value from when data(w1 to w4)\n #with the help of this index get label value\n when=tar_lab[np.argmax(lab[5:9])+5]\n\n #get index of values greater than 0.5 and get label value from it\n kind=[tar_lab[ind] for ind,ele in enumerate(lab[9:len(lab)],9) if ele>=0.5]\n\n #store the target text which is combination of sentiment,when and kind data\n #add sos at start and eos at end of text\n tar_texts+=[\"sos \"+\" \".join([senti]+[when]+kind)+\" eos\"]<\/pre>\n
#only store unique words from the input and target word lists\ninp_words = sorted(list(set(inp_words)))\nnum_inp_words = len(inp_words)\nnum_tar_words = len(tar_lab)+2\n\n#get the length of the input and the target texts which appears most frequently\nmax_inp_len = mode([len(i) for i in inp_texts])\nmax_tar_len = mode([len(i) for i in tar_texts])\n\nprint(\"number of input words : \",num_inp_words)\nprint(\"number of target words : \",num_tar_words)\nprint(\"maximum input length : \",max_inp_len)\nprint(\"maximum target length : \",max_tar_len)<\/pre>\n
<\/a><\/p>\n4) Training and Testing sets<\/h4>\n
#split the input and target text into 90:10 ratio or testing size of 10%=0.1.\nx_train,x_test,y_train,y_test=train_test_split(input_texts,target_texts,test_size=0.1,random_state=42)<\/pre>\n
5) Text Vectorization<\/h4>\n
\nL = [ \u2018on monday\u2019, \u2018free product\u2019, \u2018you are best\u2019]<\/strong>
\nTokenize all the elements of list \u2018L\u2019 and make a dictionary having keys as tokens and value as their index+1. So after the data is fit we get a dictionary as<\/p>\n
\nSo we have fit our data, now let\u2019s transform the below sentence into an integer sequence using our tokenizer.<\/p>\n
\nSo we will convert our input and target text into a sequence of integers using a vectorization technique.<\/p>\n#Use all of the words from training input and output to train the tokenizer.\ninp_tokenizer = Tokenizer()\ninp_tokenizer.fit_on_texts(x_train)\ntar_tokenizer = Tokenizer()\ntar_tokenizer.fit_on_texts(y_train)\n\n#convert text to an integer sequence where the integer represents the word index\nx_train= inp_tokenizer.texts_to_sequences(x_train)\ny_train= tar_tokenizer.texts_to_sequences(y_train)<\/pre>\n
#If the length is less than the maximum length, pad the array with 0s.\nenc_inp_data= pad_sequences(x_train, maxlen=max_inp_len, padding='post',dtype=\"float32\")\ndec_data= pad_sequences(y_train, maxlen=max_tar_len, padding='post',dtype=\"float32\")\n\n#The last word, ie 'eos,' will not be included in the decoder input data.\ndec_inp_data = dec_data[:,:-1]\n\n#decoder target data will be one time step ahead as it will not include the first initial word i.e 'sos'\ndec_tar_data = dec_data.reshape(len(dec_data),max_tar_len,1)[:,1:]<\/pre>\n
6) Build the model.<\/h4>\n
from keras import backend as K\nK.clear_session()\nlatent_dim = 500\n\n#create input object with the shape equal to the maximum number of input words\nenc_inputs = Input(shape=(max_inp_len,))\nenc_embedding = Embedding(num_inp_words+1, latent_dim)(enc_inputs)\n\n\n#create 3 stacked LSTM layer\n#1st LSTM layer keep only output\nenc_lstm1= LSTM(latent_dim, return_state=True, return_sequences=True)\nenc_outputs1, *_ = enc_lstm1(enc_embedding)\n\n#2nd LSTM layer keep only output\nenc_lstm2= LSTM(latent_dim, return_state=True, return_sequences=True)\nenc_outputs2, *_ = enc_lstm2(enc_outputs1)\n\n#3rd LSTM layer keep output as well as its states\nenc_lstm3= LSTM(latent_dim,return_sequences=True,return_state=True)\nenc_outputs3 , state_h3 , state_c3= enc_lstm3(enc_outputs2)\n\n#encoder states\nenc_states= [state_h3, state_c3]<\/pre>\n
# Decoder.\ndec_inputs = Input(shape=(None,))\ndec_emb_layer = Embedding(num_tar_words+1, latent_dim)\ndec_embedding = dec_emb_layer(dec_inputs)\n\n#initialize the LSTM layer of the decoder with the encoder's output states\ndec_lstm = LSTM(latent_dim, return_sequences=True, return_state=True)\ndec_outputs, *_ = dec_lstm(dec_embedding,initial_state=enc_states)\n\n#Attention layer\nattention =Attention()\nattn_out = attention([dec_outputs,enc_outputs3])\n\n#Merge the attention output with the decoder outputs\nmerge=Concatenate(axis=-1, name='concat_layer1')([dec_outputs,attn_out])<\/pre>\n
#fully connected Dense layer for the output\ndec_dense = Dense(num_tar_words+1, activation='softmax')\ndec_outputs = dec_dense(merge)<\/pre>\n
7) Train the model.<\/h4>\n
#Model class and model summary\nmodel = Model([enc_inputs, dec_inputs], dec_outputs)\nmodel.summary()\nplot_model(model, to_file='model_plot.png', show_shapes=True, show_layer_names=True)<\/pre>\n
<\/a><\/p>\n<\/a><\/p>\n#compile the model using RMSProp optimizer\nmodel.compile(\n optimizer=\"rmsprop\", loss=\"sparse_categorical_crossentropy\", metrics=[\"accuracy\"])\n\n#train the model with input and target data from encoder and decoder\nmodel.fit(\n [enc_inp_data, dec_inp_data],dec_tar_data,\nbatch_size=500,epochs=10)\n\n#Save model with the name as \u201cs2s\u201d\nmodel.save(\"s2s\")<\/pre>\n
<\/a><\/p>\n8) Inference Model<\/h4>\n
# encoder inference model\nlatent_dim=500\n\n#load the saved model\nmodel = models.load_model(\"s2s\")<\/pre>\n
#construct an encoder model from the output of the 6th layer of LSTM\nenc_outputs,state_h_enc,state_c_enc = model.layers[6].output\nenc_states=[state_h_enc,state_c_enc]\n\n#add input data and state data from the layer\nenc_model = Model(model.input[0],[enc_outputs]+enc_states)<\/pre>\n
# decoder inference model\n#create Input object of hidden state and cell state for decoder\ndec_state_input_h = Input(shape=(latent_dim,))\ndec_state_input_c = Input(shape=(latent_dim,))\ndec_hidden_state_input = Input(shape=(max_inp_len,latent_dim))\n\n#Get all the layers from the model\ndec_inputs = model.input[1]\ndec_emb_layer = model.layers[5]\ndec_lstm = model.layers[7]\ndec_embedding= dec_emb_layer(dec_inputs)\n\n#add input and initialize the LSTM layer with decoder\u2019s hidden and cell state\ndec_outputs2, state_h2, state_c2 = dec_lstm(dec_embedding, initial_state=[dec_state_input_h,dec_state_input_c])\n\n#Attention layer\nattention = model.layers[8]\nattn_out1 = attention([dec_outputs2,dec_hidden_state_input])\nmerge2 = Concatenate(axis=-1)([dec_outputs2, attn_out1])<\/pre>\n
#Dense layer for decoder output\ndec_dense = model.layers[10]\ndec_outputs2 = dec_dense(merge2)\n\n# Finally define the Decoder model Class\ndec_model = Model(\n[dec_inputs] + [dec_hidden_state_input,dec_state_input_h,dec_state_input_c],\n[dec_outputs2] + [state_h2, state_c2])<\/pre>\n
#create a dictionary with all indexes as key and respective target label as values\nreverse_tar_word_index = tar_tokenizer.index_word\nreverse_inp_word_index = inp_tokenizer.index_word\ntar_word_index = tar_tokenizer.word_index\nreverse_tar_word_index[0]=' '\n\ndef decode_sequence(inp_seq):\n #get the encoder outputs and states(hidden and cell) by passing the input\n sequence\n enc_out, enc_h, enc_c= enc_model.predict(inp_seq)\n\n #target sequence with starting initial word as 'sos'\n tar_seq = np.zeros((1, 1))\n tar_seq[0, 0] = tar_word_index['sos']\n\n #Stop the iteration if the iteration reaches end of the text\n stop_condition = False\n #merge every predicted word in decoded sentence\n decoded_sentence = \"\"\n while not stop_condition:\n #get predicted output words, hidden and cell state for the model\n output_words, dec_h, dec_c= dec_model.predict([tar_seq] + [enc_out,enc_h, enc_c])\n\n #Using index get the word from the dictionary\n word_index = np.argmax(output_words[0, -1, :])\n text_word = reverse_tar_word_index[word_index]\n decoded_sentence += text_word +\" \"<\/pre>\n
# Stop when we either hit max length or reach the terminal word i.e. eos.\nif text_word == \"eos\" or len(decoded_sentence) > max_tar_len:\n stop_condition = True\n\n\n#update target sequence with the current word index.\ntar_seq = np.zeros((1, 1))\ntar_seq[0, 0] = word_index\nenc_h, enc_c = dec_h, dec_c\n\n\n#return the decoded sentence string\nreturn decoded_sentence<\/pre>\n
#dict with key as label and value as target label value\nlab_val=dict((i,v) for i,v in zip(tar_lab,tar_lab_val))\n\nfor i in range(0,20,3):\n #tokenize the x_test and convert into integer sequence\n inp_x= inp_tokenizer.texts_to_sequences([x_test[i]])\n #pad array of 0's\n inp_x= pad_sequences(inp_x, maxlen=max_inp_len, padding='post')\n #reshape the input x_test\n tag=decode_sequence(inp_x.reshape(1,max_inp_len)).replace('eos','')\n print(\"Tweet:\",x_test[i])\n print(\"Predicted Hashtag:\",\" \".join([\"#\"+lab_val[i] for i in word_tokenize(tag)]))\n print(\"Actual Hashtag:\",\" \".join([\"#\"+lab_val[i] for i in y_test[i][4:-4].split(\" \")]))\n print(\"\\n\")<\/pre>\nTwitter Hashtag Prediction Output<\/h3>\n