- Querying repositories for available source packages ... Done!
The following package(s) will be installed:
- knitr [1.51]
- reticulate [1.46.0]
- rmarkdown [2.31]
These packages will be installed into "/Library/Frameworks/R.framework/Versions/4.5-arm64/Resources/library".
# Installing packages --------------------------------------------------------
[32mâś”[0m reticulate 1.46.0 [copied from cache]
[32mâś”[0m knitr 1.51 [copied from cache]
[32mâś”[0m rmarkdown 2.31 [copied from cache]
Successfully installed 3 packages in 0.27 seconds.
renv::snapshot()
The following required packages are not installed:
- MuMIn
Packages must first be installed before renv can snapshot them.
Use `]8;;x-r-run:renv::dependencies()renv::dependencies()]8;;` to see where this package is used in your project.
The following package(s) will be updated in the lockfile:
# CRAN -----------------------------------------------------------------------
- base64enc [0.1-6 -> 0.1-3]
- bslib [0.10.0 -> 0.9.0]
- cli [3.6.6 -> 3.6.5]
- digest [0.6.39 -> 0.6.38]
- fs [2.1.0 -> 1.6.6]
- highr [0.12 -> 0.11]
- htmltools [0.5.9 -> 0.5.8.1]
- insight [1.5.0 -> *]
- lifecycle [1.0.5 -> 1.0.4]
- MuMIn [1.48.19 -> *]
- nlme [3.1-168 -> *]
- png [0.1-9 -> 0.1-8]
- rappdirs [0.3.4 -> 0.3.3]
- Rcpp [1.1.1-1 -> 1.1.0]
- renv [1.2.2 -> 1.2.1]
- rlang [1.2.0 -> 1.1.6]
- tinytex [0.59 -> 0.57]
- xfun [0.57 -> 0.54]
- yaml [2.3.12 -> 2.3.10]
- glue [* -> 1.8.0]
- Lockfile written to "~/Library/CloudStorage/Dropbox/Hazel/empirical/NLP/analyses/renv.lock".
reticulate::py_install("pandas")
Using virtual environment '/Users/hazelaileenvanderwalle/.virtualenvs/r-reticulate' ...
pandas 2.3.3 OK
numpy 2.0.2 OK
nltk 3.9.2 OK
spellchecker 0.8.4 OK
All package versions OK.
Load in the data file dataMAINstudy that contains participants’ thought descriptions:
data = pd.read_csv("/Users/hazelaileenvanderwalle/Library/CloudStorage/OneDrive-Personal/work/study/PhD/_PROJECT/_framedListening_NLP/context-framed-listening/dataMAINstudy.csv")
Sentiment analysis is being conducted on music-influence mental contents (MIMC).
Create a separate dataset that only contains trials where MIMC were described, dropping all rows where “descr_THOUGHT.text” is NA.
# print number of trials with and without MIMC descriptionsnon_na_count =len(dataMIMC)na_count = data['descr_THOUGHT.text'].isna().sum()total_trials = non_na_count + na_countpercentage_with_MIMC = (non_na_count / total_trials) *100print(f"\nNumber of trials with MIMC description: {non_na_count}")
Number of trials with MIMC description: 1962
print(f"Number of trials with no MIMC description: {na_count}")
Number of trials with no MIMC description: 598
print(f"Total number of trials: {total_trials}")
Total number of trials: 2560
print(f"\n{non_na_count} out of {total_trials} trials ({percentage_with_MIMC:.1f}%) contained music-influenced mental content descriptions.")
1962 out of 2560 trials (76.6%) contained music-influenced mental content descriptions.
# Calculate number of MIMC descriptions per unique clip and context pairMIMC_per_clip_context = dataMIMC.groupby(['clip_name', 'context_word']).size().reset_index(name='MIMC_count')print(f"\nNumber of MIMC descriptions per unique clip and context pair:\n{MIMC_per_clip_context}")
Number of MIMC descriptions per unique clip and context pair:
clip_name context_word MIMC_count
0 80s_LOW_02 bar 34
1 80s_LOW_02 concert 30
2 80s_LOW_02 movie 33
3 80s_LOW_02 videogame 33
4 80s_LOW_06 bar 29
.. ... ... ...
59 Met_MED_19 videogame 31
60 Met_MED_20 bar 30
61 Met_MED_20 concert 35
62 Met_MED_20 movie 31
63 Met_MED_20 videogame 27
[64 rows x 3 columns]
# Calculate min, max, and mean MIMC counts per clip-context pairmin_mimc_count = MIMC_per_clip_context['MIMC_count'].min()max_mimc_count = MIMC_per_clip_context['MIMC_count'].max()mean_mimc_count = MIMC_per_clip_context['MIMC_count'].mean()print(f"\nAt an excerpt level, participants reported {min_mimc_count} to {max_mimc_count} thoughts (M = {mean_mimc_count:.2f}).")
At an excerpt level, participants reported 25 to 35 thoughts (M = 30.66).
Combine the clip and context values into an additional column (clip_context_PAIR). Create a clip genre column (clip_genre) to use later.
def create_clip_context_pair(row): clip_name = row['clip_name']if'bar'in row['context_word']:return'BAR-'+ clip_nameelif'videogame'in row['context_word']:return'VIDEOGAME-'+ clip_nameelif'concert'in row['context_word']:return'CONCERT-'+ clip_nameelif'movie'in row['context_word']:return'MOVIE-'+ clip_nameelse:return'NO_MATCH'dataMIMC['clip_context_PAIR'] = dataMIMC.apply(create_clip_context_pair, axis=1)# Create 'clip_genre' columndef extract_genre(clip_name):if'80s'in clip_name:return'80s'elif'Jaz'in clip_name:return'Jazz'elif'Met'in clip_name:return'Metal'elif'Ele'in clip_name:return'Electronic'else:return'UNKNOWN'dataMIMC['clip_genre'] = dataMIMC['clip_name'].apply(extract_genre)# Reorder columnscols = dataMIMC.columns.tolist()# Move 'clip_genre' to be after 'clip_name'cols.insert(cols.index('clip_name') +1, cols.pop(cols.index('clip_genre')))# Move 'clip_context_PAIR' to be after 'context_word'cols.insert(cols.index('context_word') +1, cols.pop(cols.index('clip_context_PAIR')))dataMIMC = dataMIMC[cols]# Check the dataframe by a quick re-viewprint(dataMIMC)
clip_name ... demographics.feedback
0 80s_LOW_02 ... NaN
1 Jaz_MED_07 ... NaN
2 80s_MED_08 ... NaN
3 Met_LOW_09 ... NaN
4 Met_MED_20 ... NaN
... ... ... ...
2554 Met_MED_20 ... none
2555 80s_LOW_02 ... none
2557 Jaz_MED_02 ... none
2558 Ele_MED_20 ... none
2559 Met_LOW_09 ... none
[1962 rows x 27 columns]
# Saving a .csv for the option to open and look at the full dataframebase_path ="/Users/hazelaileenvanderwalle/Library/CloudStorage/OneDrive-Personal/work/study/PhD/_PROJECT/_framedListening_NLP/context-framed-listening"dataMIMC.to_csv(f'{base_path}/NLP_outputs/dataMIMC.csv', encoding='utf-8')
Descriptive Statistics
Basic descriptive statistics on the clip-context pairings.
Create a dataframe including summary info about each clip-context stimuli pairing:
Number of participants that reported MIMC while listening
Mean MIMC and clip ratings
columns = dataMIMC.columns.tolist()[1:-1]# Drop these following columns so they don't aggregate by clip-context grouping:drop = ['clip_name', 'clip_genre', 'context_word', 'clip_context_PAIR','expName', 'File_ID', 'date', 'descr_THOUGHT.text','demographics.headphones', 'demographics.age','demographics.gender', 'demographics.livingCountry','demographics.birthCountry', 'demographics.nativeLanguage','demographics.otherLanguage', 'demographics.otherLanguageText','demographics.hearingImpariments', 'demographics.hearingImpairmentsText','demographics.education', 'demographics.musicianIdentification','demographics.feedback']# Setting up an aggregate function collectoragg_fun = {}# As we dropped trials without MIMC, we can just sum participants for MIMC occurrenceagg_fun['PROLIFIC_PID'] ='count'# Taking the mean of all columns except participant IDs and dropped columnsfor col in columns:if col notin drop and col !='PROLIFIC_PID': agg_fun[col] ='mean'# Group the dataframe by clip-context pairing, then run the aggregate functions created aboveclipContextDescrStats = dataMIMC.groupby('clip_context_PAIR').agg(agg_fun)print(clipContextDescrStats)
# Saving a .csv for the option to open and look at the full dataframeclipContextDescrStats.to_csv(f'{base_path}/NLP_outputs/clipContextDescrStats.csv', encoding='utf-8')
Check the minimum and maximum reported MIMC occurences of all clip-context stimuli pairings:
mostMIMC_ccpair = clipContextDescrStats['PROLIFIC_PID'].idxmax()leastMIMC_ccpair = clipContextDescrStats['PROLIFIC_PID'].idxmin()mostMIMC_value = clipContextDescrStats['PROLIFIC_PID'].max()leastMIMC_value = clipContextDescrStats['PROLIFIC_PID'].min()print(f"Clip-context pair with the most reported MIMC: {mostMIMC_ccpair} ({mostMIMC_value})")
Clip-context pair with the most reported MIMC: BAR-Ele_LOW_14 (35)
print(f"Clip-context pair with the least reported MIMC: {leastMIMC_ccpair} ({leastMIMC_value})")
Clip-context pair with the least reported MIMC: BAR-80s_MED_13 (25)
MIMC description word counts
Word count statistics on the raw (pre-preprocessing) thought descriptions, calculated across all MIMC trials.
# Word count on raw original descriptions (before any preprocessing)dataMIMC['word_count'] = dataMIMC['descr_THOUGHT.text'].str.split().str.len()wc_min = dataMIMC['word_count'].min()wc_max = dataMIMC['word_count'].max()wc_mean = dataMIMC['word_count'].mean()wc_sd = dataMIMC['word_count'].std()wc_median = dataMIMC['word_count'].median()print("MIMC description word counts (raw text, all trials):")
MIMC description word counts (raw text, all trials):
print(f" Min: {wc_min}")
Min: 1
print(f" Max: {wc_max}")
Max: 104
print(f" Mean: {wc_mean:.2f}")
Mean: 22.95
print(f" SD: {wc_sd:.2f}")
SD: 15.26
print(f" Median: {wc_median:.0f}")
Median: 20
# Breakdown by genre and context for referenceprint("\nMean word count by genre:")
from nltk import pos_tag, word_tokenizenltk.download('punkt_tab')
True
nltk.download('wordnet')
True
nltk.download('averaged_perceptron_tagger_eng')
True
from spellchecker import SpellCheckerspell = SpellChecker()
def find_potential_misspellings_with_context(row): misspellings_with_context = [] text = row['descr_THOUGHT.text']ifisinstance(text, str):# Tokenise the raw text tokens = word_tokenize(text)for word in tokens:# Clean the word for spell checking (lowercase and remove punctuation) cleaned_word = re.sub(r'[^a-zA-Z0-9]', '', word.lower())if cleaned_word and cleaned_word notin spell: misspellings_with_context.append((cleaned_word, text)) # Store word and original textreturn misspellings_with_context# Apply the function to each row and collect all misspellings with their contextall_misspellings_with_context = []for index, row in dataMIMC.iterrows(): misspellings_list = find_potential_misspellings_with_context(row) all_misspellings_with_context.extend(misspellings_list)# Create a DataFrame from the collected misspellings and their contextmisspellings_df = pd.DataFrame(all_misspellings_with_context, columns=['Potential Misspelling', 'Original Text'])# Display the dataFrame showing unique misspellings# (to show all occurrences, remove .drop_duplicates() <- otherwise, this shows their first occurrence context only.)print(misspellings_df.drop_duplicates(subset=['Potential Misspelling']))
Potential Misspelling Original Text
0 nt very heavy rock, not for me. somewhere that i ...
1 ve very charged, maybe you've won something or wo...
2 rememeber very bland, like lift music or something. not ...
8 doesnt Doesnt fit with what i would imagien for a vid...
9 imagien Doesnt fit with what i would imagien for a vid...
... ... ...
1204 1970s1980s I imagined this being something they'd play ba...
1205 disko I imagined people dancing and disko lights
1206 kidteen The theme song at the beginning of a kid/teen ...
1207 rihanna I imagined Rihanna performing this in a concert
1208 chaces I imagined an action movie scene where the mai...
[645 rows x 2 columns]
# Saving the full dataFrame to a CSV filemisspellings_df.to_csv(f'{base_path}/NLP_outputs/misspellings_df.csv', encoding='utf-8', index=False)# This dataFrame can be used to create your spelling dictionary.
I saved the above file out to manually check over and edit the spellings as needed. See misspelling_correction_MAINstudy.csv.
Below, apply these spelling corrections from the manually edited csv file to dataMIMC:
# Load the correction mapping from the CSV filetry: correction_df = pd.read_csv(f"{base_path}/NLP_outputs/misspelling_correction_MAINstudy.csv")# Ensure the 'correction' column is treated as strings and handle potential NaNs correction_df['correction'] = correction_df['correction'].astype(str).replace('nan', '')# Create a dictionary from the df, filtering out any remaining NaNs if necessary correction_mapping = pd.Series(correction_df.correction.values, index=correction_df.misspelling).dropna().to_dict()print("Correction mapping loaded successfully.")exceptFileNotFoundError:print("Error: misspelling_correction_MAINstudy.csv not found.") correction_mapping = {}exceptKeyError:print("Error: CSV must contain 'misspelling' and 'correction' columns.") correction_mapping = {}
Correction mapping loaded successfully.
def apply_manual_corrections(text, mapping):ifisinstance(text, str):# Clean the text (lowercase and remove punctuation) before applying corrections cleaned_text = re.sub(r'[^a-zA-Z0-9\s]', '', text.lower()) words = cleaned_text.split()# Apply corrections corrected_words = [str(mapping.get(word, word)) for word in words]return" ".join(corrected_words)returnstr(text)# Apply the manual corrections to the descr_THOUGHT.text columndataMIMC['descr_THOUGHT.text_spellchecked'] = dataMIMC['descr_THOUGHT.text'].apply(lambda x: apply_manual_corrections(x, correction_mapping))# Display the original and corrected text to check the changesprint(dataMIMC[['descr_THOUGHT.text', 'descr_THOUGHT.text_spellchecked']])
descr_THOUGHT.text descr_THOUGHT.text_spellchecked
0 kind of sad, melancholy. not happy or upbeat. ... kind of sad melancholy not happy or upbeat emo...
1 it did not sound like a video game. if anythin... it did not sound like a video game if anything...
2 overly upbeat. no real emotions, peppy. too mu... overly upbeat no real emotions peppy too much
3 very heavy rock, not for me. somewhere that i ... very heavy rock not for me somewhere that i do...
4 very charged, maybe you've won something or wo... very charged maybe you have won something or w...
... ... ...
2554 A rock band made up of teenage white kids play... a rock band made up of teenage white kids play...
2555 People in a ballroom in elegant dresses slow d... people in a ballroom in elegant dresses slow d...
2557 I imagined a jazz festival and old men on stag... i imagined a jazz festival and old men on stag...
2558 I imagined a documentary mostly about fun fact... i imagined a documentary mostly about fun fact...
2559 I imagined a wrestling show i imagined a wrestling show
[1962 rows x 2 columns]
Importing some more packages necessary for text cleaning before feeding these MIMC into our NLP models.
Stop Words
These are words we will remove from the text that won’t be accounted for in analyses.
We will define custom domain-specific stop words (e.g. music style, context cues, thought types), and get NLTK common English-language stop words.
Both preprocessing levels will have custom stop words removed, but only level 2 preprocessing for TF-IDF will has the NLTK stop words removed (as Sentence Embedding understands “the”, “is”, “and”, etc.).
Only level 2 preprocessing for TF-IDF will have lemmatisation (Sentence Embedding is able to handle word variations).
# Initialise lemmatiser and tag mappinglemmatiser = nltk.stem.WordNetLemmatizer()wn = nltk.corpus.wordnettag_map = defaultdict(lambda: wn.NOUN)tag_map['J'] = wn.ADJtag_map['V'] = wn.VERBtag_map['R'] = wn.ADV
Apply text preprocessing
LEVEL 1: Minimal preprocessing for Sentence Embedding, only removing custom stop words from spell-checked MIMC descriptions.
LEVEL 2: Aggressive preprocessing for TF-IDF, removing custom stop words, NLTK stop words, lemmatising, converting to lowercase, and removing punctuation from the spell-checked MIMC descriptions.
def to_list(arg):returnlist(arg) ifisinstance(arg, set) else argdef preprocessing_lvl1a(text, stimuli_stop_words, thought_stop_words):""" Level 1a: Minimal preprocessing for Sentence Embedding - Only removes custom domain-specific stop words - Preserves original word forms (no lemmatisation) - Keeps punctuation and capitalization for Sentence Embedding's tokeniser """ifisinstance(text, str): custom_stop_words =set(to_list(stimuli_stop_words) + to_list(thought_stop_words)) tokens = word_tokenize(text) filtered_tokens = []for word in tokens:if word.lower() notin custom_stop_words: filtered_tokens.append(word)return" ".join(filtered_tokens)return textdef preprocessing_lvl1b(text, stimuli_stop_words):""" Same as Level 1a but doesn't remove thought-related stop words """ifisinstance(text, str): custom_stop_words =set(to_list(stimuli_stop_words)) tokens = word_tokenize(text) filtered_tokens = []for word in tokens:if word.lower() notin custom_stop_words: filtered_tokens.append(word)return" ".join(filtered_tokens)return textdef preprocessing_lvl2a(text, stimuli_stop_words, thought_stop_words, nltk_stop_words, lemmatiser, tag_map):""" Level 2a: Aggressive preprocessing for TF-IDF - Removes all stop words (custom + NLTK) - Applies lemmatisation - Converts to lowercase - Removes punctuation """ifisinstance(text, str): combined_stop_words =set(to_list(stimuli_stop_words) + to_list(thought_stop_words) + to_list(nltk_stop_words)) lemmatized_stop_words =set()for sw in combined_stop_words: lemmatized_stop_words.add(lemmatiser.lemmatize(sw.lower(), wn.NOUN)) lemmatized_stop_words.add(lemmatiser.lemmatize(sw.lower(), wn.VERB)) tokens = word_tokenize(text) lemmatised_tokens = []for word, tag in pos_tag(tokens): cleaned_word = re.sub(r'[^a-zA-Z]', '', word.lower())if cleaned_word: lemmatised_word = lemmatiser.lemmatize(cleaned_word, tag_map[tag[0]])if lemmatised_word notin lemmatized_stop_words: lemmatised_tokens.append(lemmatised_word)return" ".join(lemmatised_tokens)return textdef preprocessing_lvl2b(text, stimuli_stop_words, nltk_stop_words, lemmatiser, tag_map):""" Same as Level 2a but doesn't remove thought-related stop words """ifisinstance(text, str): combined_stop_words =set(to_list(stimuli_stop_words) + to_list(nltk_stop_words)) lemmatized_stop_words =set()for sw in combined_stop_words: lemmatized_stop_words.add(lemmatiser.lemmatize(sw.lower(), wn.NOUN)) lemmatized_stop_words.add(lemmatiser.lemmatize(sw.lower(), wn.VERB)) tokens = word_tokenize(text) lemmatised_tokens = []for word, tag in pos_tag(tokens): cleaned_word = re.sub(r'[^a-zA-Z]', '', word.lower())if cleaned_word: lemmatised_word = lemmatiser.lemmatize(cleaned_word, tag_map[tag[0]])if lemmatised_word notin lemmatized_stop_words: lemmatised_tokens.append(lemmatised_word)return" ".join(lemmatised_tokens)return text# Apply Level 1a preprocessing (for Sentence Embedding)dataMIMC['textMIMC_lvl1a'] = dataMIMC['descr_THOUGHT.text_spellchecked'].apply(lambda x: preprocessing_lvl1a(x, customStopWords, thoughtStopWords))# Apply Level 1b preprocessingdataMIMC['textMIMC_lvl1b'] = dataMIMC['descr_THOUGHT.text_spellchecked'].apply(lambda x: preprocessing_lvl1b(x, customStopWords))# Apply Level 2a preprocessing (for TF-IDF)dataMIMC['textMIMC_lvl2a'] = dataMIMC['descr_THOUGHT.text_spellchecked'].apply(lambda x: preprocessing_lvl2a(x, customStopWords, thoughtStopWords, nltk_stop_words, lemmatiser, tag_map))# Apply Level 2b preprocessingdataMIMC['textMIMC_lvl2b'] = dataMIMC['descr_THOUGHT.text_spellchecked'].apply(lambda x: preprocessing_lvl2b(x, customStopWords, nltk_stop_words, lemmatiser, tag_map))# Display all text processing stagesprint(dataMIMC[['descr_THOUGHT.text','descr_THOUGHT.text_spellchecked','textMIMC_lvl1a','textMIMC_lvl1b','textMIMC_lvl2a','textMIMC_lvl2b']])
descr_THOUGHT.text ... textMIMC_lvl2b
0 kind of sad, melancholy. not happy or upbeat. ... ... kind sad melancholy happy upbeat emotionally c...
1 it did not sound like a video game. if anythin... ... sound like anything maybe end credit something...
2 overly upbeat. no real emotions, peppy. too mu... ... overly upbeat real emotion peppy much
3 very heavy rock, not for me. somewhere that i ... ... heavy somewhere belong enjoy
4 very charged, maybe you've won something or wo... ... charge maybe something battle hype possibly en...
... ... ... ...
2554 A rock band made up of teenage white kids play... ... make teenage white kid garage
2555 People in a ballroom in elegant dresses slow d... ... people ballroom elegant dress slow dance floor
2557 I imagined a jazz festival and old men on stag... ... imagine festival old men stage
2558 I imagined a documentary mostly about fun fact... ... imagine documentary mostly fun fact historic i...
2559 I imagined a wrestling show ... imagine wrestling show
[1962 rows x 6 columns]
# Save out for analyses and inspectiondataMIMC.to_csv(f'{base_path}/NLP_outputs/dataMIMC_preprocessed.csv', encoding='utf-8')print("\nPreprocessing complete")
Preprocessing complete
print("- 'textMIMC_lvl1a': Minimal filtering for Sentence Embedding (custom stop words only)")
- 'textMIMC_lvl1a': Minimal filtering for Sentence Embedding (custom stop words only)
print("- 'textMIMC_lvl1b': Minimal filtering for Sentence Embedding (stimuli stop words only)")
- 'textMIMC_lvl1b': Minimal filtering for Sentence Embedding (stimuli stop words only)
print("- 'textMIMC_lvl2a': Full preprocessing for TF-IDF (all stop words + lemmatisation)")
- 'textMIMC_lvl2a': Full preprocessing for TF-IDF (all stop words + lemmatisation)
print("- 'textMIMC_lvl2b': Full preprocessing for TF-IDF (stimuli+NLTK stop words + lemmatisation)")
- 'textMIMC_lvl2b': Full preprocessing for TF-IDF (stimuli+NLTK stop words + lemmatisation)
dataMIMC (saved out as "dataMIMC_preprocessed.csv") now contains the preprocessed data from individual participants (one row = one MIMC) with the two levels of filtering: MIMCdata_prepLVL1 and MIMCdata_prepLVL2. We will refer to this file for the NLP analyses.
Creating Combined combMIMC
Here, we are combining the preprocessed MIMC descriptions into singular aggregated documents grouped by stimuli condition i.e. specific clip and context pairing. This produces 64 document combinations (16 clips Ă— 4 contexts).
Where the individual preprocessed MIMC descriptions can be analysed to measure semantic similarity between individual thought descriptions, combMIMC can be analysed to measure semantic similarity between aggregated representations of conditions.
def build_combMIMC(dataMIMC, text_col):""" Aggregate individual MIMC descriptions by clip-context pair. Returns a DataFrame with one row per pair containing: - the aggregated NLP text column (joined with 'endofinput' separators) - text_original: the spell-checked original descriptions joined with ' | ' separators, retained for human-readable reference in downstream analyses - metadata columns (clip_name, context_word, genre_code, etc.) """ rows = []for idStimPair in np.unique(dataMIMC['clip_context_PAIR']): stimPairMask = dataMIMC['clip_context_PAIR'] == idStimPair filt = dataMIMC[stimPairMask]iflen(filt) ==0:continue first_row = filt.iloc[0] idClip = first_row['clip_name'] idContext = first_row['context_word']# Aggregate NLP-preprocessed text (existing behaviour) joined_nlp =" endofinput ".join(str(v) for v in filt[text_col].values) +" endofinput "# Aggregate original spell-checked text for human-readable reference# Each participant's response is separated by ' | ' so the column remains# readable when inspecting the CSV or displaying pairs in analyses orig_col_src ='descr_THOUGHT.text_spellchecked' joined_orig =" | ".join(str(v) for v in filt[orig_col_src].valuesifisinstance(v, str) and v.strip())# Derive genre code from clip name prefix prefix = idClip[0:3] genre_map = {'80s': '80s', 'Jaz': 'Jaz', 'Met': 'Met', 'Ele': 'Ele'} genre_code = genre_map.get(prefix, 'UNK') rows.append({'context_word': idContext,'genre_code': genre_code,'clip_name': idClip,'ClipContext_pair': idStimPair,'GenreContext_pair': idStimPair[0:3] +"_"+ prefix, text_col: joined_nlp,'text_original': joined_orig, })return pd.DataFrame(rows)# Build all four combMIMC DataFramescombMIMC_lvl1a = build_combMIMC(dataMIMC, 'textMIMC_lvl1a')combMIMC_lvl1b = build_combMIMC(dataMIMC, 'textMIMC_lvl1b')combMIMC_lvl2a = build_combMIMC(dataMIMC, 'textMIMC_lvl2a')combMIMC_lvl2b = build_combMIMC(dataMIMC, 'textMIMC_lvl2b')# Save — text_original is now included in every filecombMIMC_lvl1a.to_csv(f'{base_path}/NLP_outputs/combMIMC_lvl1a.csv', encoding='utf-8', index=False)combMIMC_lvl1b.to_csv(f'{base_path}/NLP_outputs/combMIMC_lvl1b.csv', encoding='utf-8', index=False)combMIMC_lvl2a.to_csv(f'{base_path}/NLP_outputs/combMIMC_lvl2a.csv', encoding='utf-8', index=False)combMIMC_lvl2b.to_csv(f'{base_path}/NLP_outputs/combMIMC_lvl2b.csv', encoding='utf-8', index=False)print(f"Created {len(combMIMC_lvl1a)} documents (Clip-Context combinations), combining Level 1a preprocessed MIMC descriptions")
clip_name ... text_original
0 80s_LOW_02 ... kind of sad melancholy not happy or upbeat emo...
1 80s_LOW_06 ... the music felt really 80s and so it was an ima...
2 80s_MED_08 ... i think about an old 80s movie that takes plac...
3 80s_MED_13 ... i could only think about sitting on the couch ...
4 Ele_LOW_09 ... this made me think about europe or a specific ...
[5 rows x 5 columns]
clip_name ... text_original
0 80s_LOW_02 ... kind of sad melancholy not happy or upbeat emo...
1 80s_LOW_06 ... the music felt really 80s and so it was an ima...
2 80s_MED_08 ... i think about an old 80s movie that takes plac...
3 80s_MED_13 ... i could only think about sitting on the couch ...
4 Ele_LOW_09 ... this made me think about europe or a specific ...
[5 rows x 5 columns]
clip_name ... text_original
0 80s_LOW_02 ... kind of sad melancholy not happy or upbeat emo...
1 80s_LOW_06 ... the music felt really 80s and so it was an ima...
2 80s_MED_08 ... i think about an old 80s movie that takes plac...
3 80s_MED_13 ... i could only think about sitting on the couch ...
4 Ele_LOW_09 ... this made me think about europe or a specific ...
[5 rows x 5 columns]
clip_name ... text_original
0 80s_LOW_02 ... kind of sad melancholy not happy or upbeat emo...
1 80s_LOW_06 ... the music felt really 80s and so it was an ima...
2 80s_MED_08 ... i think about an old 80s movie that takes plac...
3 80s_MED_13 ... i could only think about sitting on the couch ...
4 Ele_LOW_09 ... this made me think about europe or a specific ...
[5 rows x 5 columns]
print("\nAll combMIMC files now include 'text_original': the concatenated spell-checked")
All combMIMC files now include 'text_original': the concatenated spell-checked
print("original descriptions separated by ' | ' for human-readable reference.")
original descriptions separated by ' | ' for human-readable reference.
Text preprocessing is complete!
The saved out files combMIMC_lvl1a.csv, combMIMC_lvl1b.csv, combMIMC_lvl2a.csv, and combMIMC_lvl2b.csv will also be used alongside the dataMIMC_preprocessed.csv for the different levels of analysis in subsequent NLP analyses.
(Reminder: LVL1 & LVL2 have thought words removed, LVL1A & LVL2A have thought words kept in.)